Bionic robot interaction behavior control method and device, bionic robot and medium

Abstract

The application relates to a bionic robot interactive behavior control method and device, a bionic robot and a medium, wherein the method comprises the following steps: in a scene in which the bionic robot interacts with an interactive object, acquiring interactive scene information and pose information of the bionic robot; obtaining an interactive behavior instruction according to the interactive scene information and the pose information and a preset behavior generation model, the behavior generation model being obtained by training an initial behavior generation model according to sample interactive scene information, sample pose information of the bionic robot and a gold standard interactive instruction, the gold standard interactive instruction being generated based on sample video data of the bionic object of the bionic robot interacting with the interactive object, and interactive behavior data of the bionic object from the perspective of the bionic object being reconstructed; and generating a corresponding interactive behavior based on the interactive behavior instruction; thus, the bionic robot can make continuous and lively interactive behaviors according to specific conditions.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method, device, bionic robot, and medium for controlling the interactive behavior of a bionic robot. Background Technology

[0002] For biomimetic robots, in the process of interacting with interactive objects, they need to have biological-like natural interaction capabilities to respond to the gestures and body interactions of interactive objects in real life scenarios and provide emotional value to interactive objects.

[0003] Bionic robots in related technologies are mainly based on classification models, which visually recognize the behavior of interactive objects as fixed intentions, and then perform corresponding interactive behaviors based on the recognized intentions and a pre-programmed action library.

[0004] However, in traditional technologies, bionic robots suffer from problems such as inconsistent and unrealistic interactive behaviors when interacting with interactive objects. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, bionic robot, and medium for controlling the interactive behavior of a bionic robot, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for controlling the interactive behavior of a biomimetic robot, the method comprising:

[0007] In scenarios where bionic robots interact with interactive objects, information about the interaction scenario and the pose information of the bionic robot are obtained.

[0008] Based on the interaction scene information, pose information, and the preset behavior generation model, the interaction behavior instructions are obtained. The behavior generation model is obtained by training the initial behavior generation model based on the sample interaction scene information, the sample pose information of the bionic robot, and the gold standard interaction instructions. The gold standard interaction instructions are generated by reconstructing the interaction behavior data of the bionic object from the perspective of the bionic object based on the sample video data of the interaction between the bionic robot and the bionic object.

[0009] Based on the interactive behavior instructions, the corresponding interactive behavior is generated.

[0010] In one embodiment, the method further includes:

[0011] Based on a preset 3D scene construction algorithm, a 3D scene corresponding to the sample video data is constructed.

[0012] In a 3D scene, a preset depth estimation algorithm is used to reconstruct the interaction behavior and trajectory of the bionic object and the interactive object in the sample video data;

[0013] Simulate and render the reconstructed interactive behaviors and trajectories to generate interactive behavior data of the bionic object from the perspective of the bionic object.

[0014] By using keypoint detection algorithms to process interactive behavior data, the gold standard interactive commands are obtained.

[0015] In one embodiment, a keypoint detection algorithm is used to process the interaction behavior data to obtain the gold standard interaction instructions, including:

[0016] Key point detection algorithms are used to determine the joints of biomimetic objects;

[0017] By using a motion redirection algorithm, interactive behavior data is mapped to the joints of a biomimetic object to obtain the gold standard interactive commands.

[0018] In one embodiment, the reconstructed interactive behavior and the reconstructed interactive trajectory are simulated and rendered to generate interactive behavior data of the biomimetic object from the perspective of the biomimetic object, including:

[0019] Construct virtual biomimetic objects of the same size as the biomimetic objects in virtual interactive scenarios;

[0020] Based on the reconstructed interactive behavior and the reconstructed interactive trajectory, the position and posture of the virtual bionic object at each point in time are adjusted to generate interactive behavior data of the bionic object from the perspective of the bionic object.

[0021] In one embodiment, the behavior generation model includes a ViT model, a Transformer temporal information encoding model, and a Diffusion policy model; based on the interaction scene information, pose information, and the preset behavior generation model, interaction behavior instructions are obtained, including:

[0022] The ViT model is used to process the sample interaction scene information and sample pose information to obtain the corresponding feature vectors;

[0023] The Transformer temporal information encoding model is used to determine the temporal relationship between feature vectors and obtain the encoded feature vectors corresponding to the feature vectors.

[0024] The Diffusion policy model is used to sample and denoise the encoded feature vector to obtain interactive behavior instructions.

[0025] In one embodiment, the method further includes:

[0026] Determine the processor type for the bionic robot;

[0027] The trained behavior generation model is converted into a new format based on the processor type to obtain the behavior generation model.

[0028] Secondly, this application also provides a biomimetic robot interactive behavior control device, the device comprising:

[0029] The data acquisition module is used to acquire interaction scene information and the pose information of the bionic robot in scenarios where the bionic robot interacts with the interactive object.

[0030] The instruction generation module is used to obtain interactive behavior instructions based on interactive scene information, pose information, and a preset behavior generation model. The behavior generation model is obtained by training an initial behavior generation model based on sample interactive scene information, sample pose information of the bionic robot, and gold standard interactive instructions. The gold standard interactive instructions are generated from the interactive behavior data of the bionic object from the perspective of the reconstructed bionic object based on sample video data of the interaction between the bionic robot's bionic object and the interactive object.

[0031] The behavior control module is used to generate corresponding interactive behaviors based on interactive behavior commands.

[0032] Thirdly, this application also provides a bionic robot, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] In scenarios where bionic robots interact with interactive objects, information about the interaction scenario and the pose information of the bionic robot are obtained.

[0034] Based on the interaction scene information, pose information, and the preset behavior generation model, the interaction behavior instructions are obtained. The behavior generation model is obtained by training the initial behavior generation model based on the sample interaction scene information, the sample pose information of the bionic robot, and the gold standard interaction instructions. The gold standard interaction instructions are generated by reconstructing the interaction behavior data of the bionic object from the perspective of the bionic object based on the sample video data of the interaction between the bionic robot and the bionic object.

[0035] Based on the interactive behavior instructions, the corresponding interactive behavior is generated.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0037] In scenarios where bionic robots interact with interactive objects, information about the interaction scenario and the pose information of the bionic robot are obtained.

[0038] Based on the interaction scene information, pose information, and the preset behavior generation model, the interaction behavior instructions are obtained. The behavior generation model is obtained by training the initial behavior generation model based on the sample interaction scene information, the sample pose information of the bionic robot, and the gold standard interaction instructions. The gold standard interaction instructions are generated by reconstructing the interaction behavior data of the bionic object from the perspective of the bionic object based on the sample video data of the interaction between the bionic robot and the bionic object.

[0039] Based on the interactive behavior instructions, the corresponding interactive behavior is generated.

[0040] Fifthly, this application also provides a computer program product comprising a computer program that, when executed by a processor, performs the following steps:

[0041] In scenarios where bionic robots interact with interactive objects, information about the interaction scenario and the pose information of the bionic robot are obtained.

[0042] Based on the interaction scene information, pose information, and the preset behavior generation model, the interaction behavior instructions are obtained. The behavior generation model is obtained by training the initial behavior generation model based on the sample interaction scene information, the sample pose information of the bionic robot, and the gold standard interaction instructions. The gold standard interaction instructions are generated by reconstructing the interaction behavior data of the bionic object from the perspective of the bionic object based on the sample video data of the interaction between the bionic robot and the bionic object.

[0043] Based on the interactive behavior instructions, the corresponding interactive behavior is generated.

[0044] The aforementioned bionic robot interactive behavior control method, device, bionic robot, and medium, in the scenario of bionic robot interacting with an interactive object, first acquires the interaction scene information and the pose information of the bionic robot; then, based on the interaction scene information, pose information, and a preset behavior generation model, obtains interactive behavior commands; finally, based on the interactive behavior commands, generates corresponding interactive behaviors. Since the behavior generation model is trained on the initial behavior generation model based on sample interaction scene information, sample pose information of the bionic robot, and gold standard interactive commands, and the gold standard interactive commands are generated based on sample video data of the interaction between the bionic robot and the interactive object, reconstructing the interactive behavior data of the bionic object from the perspective of the bionic object, the behavior generation model can learn the interactive behavior patterns between the bionic object and the interactive object in real-world scenarios, improving the adaptability of the bionic robot in real-world scenarios, and generating diverse interactive behavior commands according to the specific circumstances of the real-world scenario, enabling the bionic robot to perform continuous and vivid interactive behaviors. Attached Figure Description

[0045] Figure 1Flowcharts of a biomimetic robot interactive behavior control method provided in some embodiments of this application;

[0046] Figure 2 A flowchart of obtaining gold standard interactive instructions is provided for some embodiments of this application;

[0047] Figure 3 A flowchart for obtaining gold standard interactive instructions is provided for some embodiments of this application;

[0048] Figure 4 A flowchart illustrating the interactive behavior data of a biomimetic object from the perspective of generating a biomimetic object, provided in some embodiments of this application;

[0049] Figure 5 Flowcharts for obtaining interactive behavior instructions provided in some embodiments of this application;

[0050] Figure 6 A flowchart illustrating the behavior generation model provided in some embodiments of this application;

[0051] Figure 7 Structural block diagram of a biomimetic robot interactive behavior control device provided in some embodiments of this application;

[0052] Figure 8 This is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The bionic robot interactive behavior control method provided in this application embodiment can be applied to bionic robots. Specifically, the controller of the bionic robot can execute the bionic robot interactive behavior control method provided in this application embodiment. In a scenario where the bionic robot interacts with an interactive object, the controller first acquires the interaction scene information and the pose information of the bionic robot; then, based on the interaction scene information, pose information, and a preset behavior generation model, it obtains interactive behavior instructions; finally, based on the interactive behavior instructions, it generates corresponding interactive behaviors. Since the behavior generation model is trained on an initial behavior generation model based on sample interaction scene information, sample pose information of the bionic robot, and gold standard interactive instructions, and the gold standard interactive instructions are generated based on sample video data of the interaction between the bionic robot and the interactive object, reconstructing the interactive behavior data of the bionic object from the perspective of the bionic object, the behavior generation model can learn the interactive behavior patterns of the bionic object and the interactive object in real scenarios, improve the adaptability of the bionic robot in real scenarios, and generate diverse interactive behavior instructions according to the specific situation of the real scenario, so that the bionic robot can perform continuous and vivid interactive behaviors.

[0055] In one embodiment, such as Figure 1 As shown, this method is illustrated using an example of its application in the controller of a biomimetic robot. In this embodiment, the method includes the following steps:

[0056] Step 102: In the scenario where the bionic robot interacts with the interactive object, obtain the interaction scenario information and the pose information of the bionic robot.

[0057] In this context, the biomimetic robot can be a quadrupedal robot dog, a humanoid robot, or other robots that mimic the form and behavior of living organisms. In this embodiment, a quadrupedal robot dog is used as an example; it has movable limbs and can simulate a dog's walking, running, and jumping movements. The interaction object is the object interacting with the biomimetic robot, that is, a human individual interacting with the biomimetic robot, but it can also be other organisms or objects.

[0058] Interactive scene information includes video stream data of the surrounding environment. This video stream contains obstacle feature information (such as the shape, size, and position of obstacles), distance information between the bionic robot and the interactive object, scene lighting information, and object distribution information. This information can be acquired using devices such as cameras and depth sensors mounted on the bionic robot. The bionic robot's pose information includes its position in three-dimensional space (such as x, y, and z coordinates in a Cartesian coordinate system) and its orientation (such as rotation angles around the x, y, and z axes). For quadrupedal robot dogs, this also includes the angle information of each joint, which can be acquired using devices such as inertial measurement units (IMUs) and joint angle sensors installed on the quadrupedal robot dog.

[0059] Optionally, the camera on the bionic robot can be used to collect video stream data of the surrounding environment as interactive scene information, the position and attitude information of the robot can be obtained through the inertial measurement unit, and the angle information of each joint can be obtained through the joint angle sensor as the pose information of the bionic robot.

[0060] Step 104: Obtain interactive behavior instructions based on the interactive scene information, pose information, and preset behavior generation model.

[0061] The behavior generation model is trained using sample interaction scene information, sample pose information of the bionic robot, and gold standard interaction commands. The gold standard interaction commands are generated from sample video data of the interaction between the bionic robot and the interactive object, reconstructing the interactive behavior data of the bionic object from the perspective of the bionic object. The interaction behavior commands are the output of the behavior generation model and are used to control the actuators of each joint of the bionic robot, enabling the robot to perform corresponding interactive behaviors. For example, for a quadruped robot dog, the interaction behavior commands can include information such as the rotation angle, movement speed, and walking direction of each joint.

[0062] Optionally, a convolutional neural network can be used to extract features from the interactive scene information, a multilayer perceptron can be used to process the pose information of the bionic robot, the two types of processed information can be merged and input into a recurrent neural network to process the temporal information, and finally the interactive behavior instructions can be output through a fully connected layer.

[0063] Specifically, a behavior generation model integrating convolutional neural networks, long short-term memory networks, and policy gradient networks can be used to obtain interactive behavior instructions based on interaction scene information, pose information, and a pre-set behavior generation model. Specifically, for sample interaction scene information (such as images), a pre-trained CNN model (such as ResNet) can be used for feature extraction to obtain high-level semantic features of the image; for sample pose information, a simple fully connected neural network can be constructed for feature processing. For example, the sample pose information can be normalized first, then input into a fully connected layer, and a pose feature vector can be obtained through a non-linear activation function (such as ReLU); then the scene features and pose feature vectors are concatenated to obtain a fused feature vector; then the fused feature vectors are arranged into a sequence in chronological order, and this sequence is input into a long-term neural network. In a short-term memory network, the long short-term memory network outputs the hidden state at each time step. These hidden states contain the temporal information of the input sequence. Since the hidden state at the last time step integrates the temporal features of the entire sequence, it can be used as the encoded feature vector. A policy gradient network is then constructed, and the encoded feature vector is input into it. During the training phase, policy gradient algorithms (such as proximal policy optimization, deep deterministic policy gradient, etc.) are used for training. During the inference phase, the encoded feature vector is input into the trained policy gradient network, and the network directly outputs the interactive behavior instructions.

[0064] Step 106: Generate the corresponding interactive behavior based on the interactive behavior instructions.

[0065] Optionally, interactive behavior commands can be sent to the actuators of each joint of the bionic robot through the controller to control the movement of the joints, thereby enabling the bionic robot to perform corresponding interactive behaviors.

[0066] The aforementioned bionic robot interactive behavior control method, in the scenario of bionic robot interacting with an interactive object, first acquires the interaction scene information and the pose information of the bionic robot; then, based on the interaction scene information, pose information, and a preset behavior generation model, it obtains interactive behavior commands; finally, based on the interactive behavior commands, it generates corresponding interactive behaviors. Since the behavior generation model is trained on the initial behavior generation model based on sample interaction scene information, sample pose information of the bionic robot, and gold standard interactive commands, and the gold standard interactive commands are generated based on sample video data of the interaction between the bionic robot and the interactive object, reconstructing the interactive behavior data of the bionic object from the perspective of the bionic object, the behavior generation model can learn the interactive behavior patterns between the bionic object and the interactive object in real-world scenarios, improving the adaptability of the bionic robot in real-world scenarios, and generating diverse interactive behavior commands according to the specific circumstances of the real-world scenario, so that the bionic robot can perform continuous and vivid interactive behaviors.

[0067] In one embodiment, such as Figure 2 As shown, the method also includes:

[0068] Step 202: Construct a 3D scene corresponding to the sample video data based on a preset 3D scene construction algorithm.

[0069] Optionally, the MonST3R algorithm can be used to generate time-varying dynamic point clouds, camera poses, and depth maps for each frame from dynamic video, thereby constructing a 3D scene. Specifically, pairwise point map estimation is performed first: for sample video data (videos of dynamic scenes captured by ordinary cameras, i.e., monocular videos), a sliding window mechanism is used to analyze the data within a fixed-size time window (such as N consecutive frames). Each pair of image frames within the selected window is input into a finely tuned MonST3R model (based on the Vision Transformer architecture). The model regresses the 3D coordinates of each pixel in the first camera coordinate system for each pixel of the two frames, generating paired, aligned point maps. Global optimization and pose estimation are then performed: optical flow estimation algorithms (such as RAFT) are used to calculate pixel-level motion between frames, static regions in the video frames are identified, and robust estimation algorithms such as RANSAC and PnP are used to calculate the relative pose of each camera frame. All paired point maps are then transformed to a unified global coordinate system, and constraints such as trajectory smoothing loss and optical flow projection loss are introduced to optimize the entire point cloud and camera trajectory, outputting a complete, time-varying dynamic point cloud to complete the 3D scene construction.

[0070] In another embodiment, the 3D scene corresponding to the sample video data can also be constructed based on the COLMAP algorithm and the OpenMVG + OpenMVS algorithm. For example, taking the OpenMVG + OpenMVS algorithm as an example, the camera pose and sparse 3D point cloud can be estimated by the OpenMVG algorithm, and then a dense 3D point cloud can be generated by the OpenMVS algorithm based on the processing result of the OpenMVG algorithm.

[0071] Step 204: In a 3D scene, a preset depth estimation algorithm is used to reconstruct the interaction behavior and interaction trajectory of the bionic object and the interactive object in the sample video data.

[0072] Interactive behaviors can be the various actions performed by the bionic object (such as a real dog) during interaction with the interactive object, such as the dog running towards the user when the user waves. Interactive trajectories can be the movement paths of the bionic object and the interactive object in a three-dimensional scene during the interaction, such as the path the dog takes as it circles the user, or the user's walking trajectory while interacting with the dog. Interactive behaviors and trajectories can fully describe the interaction process between the bionic object and the user, providing detailed information for the subsequent generation of gold-standard interaction commands, enabling the bionic robot to learn realistic and natural interaction patterns.

[0073] Optionally, in the constructed 3D scene, object detection algorithms (such as YOLO, Faster R-CNN, or 3D object detection algorithms) are used to detect the position, orientation, and bounding box of the bionic object and the user in each frame. Tracking algorithms (such as SORT and ByteTrack) are used to track the position, orientation, and bounding box of the bionic object and the user between consecutive frames. Combining the depth information and target position of each frame, the positions of the bionic object and the user in 3D space are connected in chronological order to generate interactive behaviors and interactive trajectories in 4D time sequence.

[0074] In another embodiment, the interaction behaviors and trajectories of bionic objects and interactive objects in sample video data can also be reconstructed based on an end-to-end deep learning approach. Specifically, an end-to-end deep learning model can be trained to directly learn the interaction behaviors and trajectories of bionic objects and interactive objects from sample video data. This model can take a sequence of video frames as input and output representations of the interaction behaviors and trajectories. For example, a combined architecture based on convolutional neural networks and recurrent neural networks can be used, where the convolutional neural network is used to extract spatial features of video frames, and the recurrent neural network is used to process the event sequence information of video frames.

[0075] Step 206: Simulate and render the reconstructed interactive behavior and reconstructed interactive trajectory to generate interactive behavior data of the bionic object from the perspective of the bionic object.

[0076] Simulation rendering is the process of visualizing reconstructed interactive behaviors and trajectories within a 3D scene to generate realistic images or videos. This process requires considering factors such as lighting, materials, and shadows to simulate realistic environmental effects.

[0077] Optionally, 3D modeling and rendering software such as Blender and Unity can be used to render the reconstructed 3D model and interactive animation, generating interactive behavior data from the perspective of the biomimetic object.

[0078] Step 208: Process the interaction behavior data using a key point detection algorithm to obtain the gold standard interaction instructions.

[0079] Among them, key point detection algorithms are used to extract representative key points from interaction behavior data, such as the joints of bionic objects and the key points of a user's hand.

[0080] Optionally, a key point detection algorithm can be used to determine the joints of the biomimetic object; then, based on the coordinate and motion information of the key points, gold standard interaction commands can be generated, such as the rotation angle and motion speed of the joints.

[0081] In another embodiment, gold standard interaction instructions can also be generated based on instructions from a generative adversarial network (GAN). Specifically, a GAN can be constructed, including a generator and a discriminator. The generator takes interaction behavior data as input and outputs gold standard interaction instructions. The discriminator is used to determine whether the generated instructions conform to the real interaction standard. Through adversarial training, the performance of the generator and discriminator is continuously optimized, enabling the generator to generate more realistic and effective gold standard interaction instructions.

[0082] In this embodiment, by constructing a 3D scene, estimating depth, and performing simulation rendering, more realistic and accurate interactive behavior data from the perspective of the bionic object can be generated, enabling the behavior generation model trained subsequently to learn interaction patterns that are closer to real-world situations. In addition, the gold standard interaction instructions are generated based on real interaction behaviors and trajectories, which can cover more interaction scenarios and variations, improve the generalization ability of the behavior generation model, and enable the bionic robot to perform reasonable, natural, and smooth interactive behaviors in different real-world scenarios.

[0083] In one embodiment, such as Figure 3 As shown, the interaction behavior data is processed using a keypoint detection algorithm to obtain the gold standard interaction commands, including:

[0084] Step 302: Use a key point detection algorithm to determine the joints of the biomimetic object.

[0085] Optionally, a suitable 3D keypoint detection model can be selected first. This could involve combining 2D keypoint detection algorithms like OpenPose and HRNet with enhancement algorithms like VideoPose3D to map 2D keypoints to 3D coordinates. Alternatively, an end-to-end 3D keypoint detection algorithm like Volumetric Heatmap can be used. The model can be trained using an image dataset containing images of different poses of the bionic object, enabling it to accurately identify the joints of the bionic object's limbs, such as the positions of the shoulder, elbow, hip, and knee joints. Each frame of the video is then input into the trained keypoint detection model, which outputs the coordinates of the bionic object's limb joints within the image. For example, in one frame, the pixel coordinates of the detected left forelimb shoulder joint are (x1, y1, z1), and the elbow joint coordinates are (x2, y2, z2), etc. These coordinates are then aligned with the reconstructed dynamic environment.

[0086] Step 304: Using a motion redirection algorithm, the interactive behavior data is mapped to the joints of the bionic object to obtain the gold standard interactive instructions.

[0087] Optionally, the motion retargeting algorithm can be a motion retargeting algorithm. First, the skeletal structures of the biomimetic object and the biomimetic robot are defined separately, determining the name and connection relationship of each joint. For example, the skeletal structure of a real dog may include joints in the head, neck, and limbs, and the skeletal structure of the robot dog also has corresponding joints. Then, the three-dimensional coordinates of the detected joints of the real dog's limbs (obtained through depth information and image coordinate transformation) are mapped onto the skeletal structure of the robot dog. Based on the similarity between the skeletal structures of the real dog and the robot dog, the rotation and translation parameters of the corresponding joints are calculated, enabling the robot dog's joints to mimic the limb movements of a real dog. For example, if the left forelimb elbow of a real dog is bent at 30 degrees, through mapping calculation, the same bending angle is applied to the left forelimb elbow joint of the robot dog. In this way, the gold standard interaction command for the left forelimb elbow joint in this scenario can also be obtained simultaneously.

[0088] In this embodiment, the motion redirection algorithm maps the interactive behavior data to the joints of the bionic object to obtain the gold standard interactive instructions, which can provide accurate control basis for the bionic robot to imitate the behavior of the bionic object.

[0089] In one embodiment, such as Figure 4 As shown, the reconstructed interactive behavior and reconstructed interactive trajectory are simulated and rendered to generate interactive behavior data of the bionic object from the perspective of the bionic object, including:

[0090] Step 402: Construct a virtual bionic object with the same size as the bionic object in the virtual interactive scene.

[0091] Optionally, the reconstructed 3D environment can be imported into simulation software (such as Unity, Unreal Engine, etc.) in a suitable file format (such as OBJ, FBX, etc.). The simulation software then constructs a virtual scene based on the point cloud and texture information of the 3D environment.

[0092] Step 404: Based on the reconstructed interactive behavior and the reconstructed interactive trajectory, adjust the position and posture of the virtual bionic object at each time point to generate interactive behavior data of the bionic object from the perspective of the bionic object.

[0093] Optionally, a virtual model similar in shape and size to the biomimetic object can be placed in the simulation environment first. The model's position and orientation are then adjusted based on the reconstructed interaction trajectory to ensure its movement matches reality. A virtual camera is then positioned at the virtual model's viewpoint to simulate the biomimetic object's perspective, with the camera's orientation and orientation matching those of the biomimetic object. At each time point, the simulation software renders the virtual scene based on the virtual camera's position and orientation, generating interactive behavior data from the biomimetic object's perspective—essentially simulating the image data captured by the virtual camera.

[0094] In this embodiment, by generating interactive behavior data from the perspective of the biomimetic object, developers or researchers can more intuitively understand the actual feelings and perspectives of the biomimetic object during the interaction process, providing a more realistic reference for further optimizing human-computer interaction design.

[0095] In one embodiment, such as Figure 5 As shown, the behavior generation model includes the ViT model, the Transformer temporal information encoding model, and the Diffusion policy model; based on the interaction scene information, pose information, and the preset behavior generation model, the interaction behavior instructions are obtained, including:

[0096] Step 502: The ViT model is used to process the sample interaction scene information and sample pose information to obtain the corresponding feature vectors.

[0097] Among them, the feature vector is a numerical vector obtained by the ViT model after processing the sample interaction scene information and sample pose information. These vectors contain the key features of the input information and are a compact representation of the original information. They help to transform complex interaction scene information and pose information into feature representations that are easy to process later, extract important features from the input information, reduce information redundancy, and provide more effective input for subsequent models.

[0098] Optionally, the sample interaction scene information (such as scene information in image form) can be first segmented into fixed-size patches, and each patch can be flattened into a one-dimensional vector. For the sample pose information, preprocessing operations such as normalization can be performed to match the feature dimension of the image patch. Then, position encoding is added to each patch and pose information to preserve its position information in the original data. The preprocessed data is then input into the Transformer encoder of the ViT model, which consists of multiple multi-head self-attention layers and feedforward neural network layers. In the multi-head self-attention layers, the ViT model calculates the attention score between each element and other elements to capture global relationships. Through multiple attention calculations (multi-head), features can be extracted from different representation subspaces. The feedforward neural network layer performs a non-linear transformation on the output of the attention layer to further extract features. Finally, after processing by multiple layers of encoders, the ViT model outputs a feature vector, and the output of the last encoder layer can be selected as the final feature vector.

[0099] Step 504: Using the Transformer temporal information encoding model, determine the temporal relationship between feature vectors and obtain the encoded feature vectors corresponding to the feature vectors.

[0100] Optionally, the feature vectors obtained in step 502 are first arranged in chronological order to form a sequence, and then positional encoding is added to the sequence to represent the position of each feature vector in the time series. The sequence is then input into the Transformer temporal information encoding model, which also includes a multi-head self-attention layer and a feedforward neural network layer. In the multi-head self-attention layer, the Transformer temporal information encoding model calculates the attention scores between feature vectors at different time points to capture temporal dependencies. The feedforward neural network layer performs a nonlinear transformation on the output of the attention layer to enhance the feature representation. Finally, after multiple layers of encoding, the encoded feature vectors are obtained, which contain temporal relationship information between feature vectors.

[0101] Step 506: The Diffusion policy model is used to sample and denoise the encoded feature vector to obtain the interactive behavior instructions.

[0102] Optionally, one can start with the encoded feature vector and gradually add noise to it to transform it into a noise distribution; then train the Diffusion policy model to learn how to recover the original encoded feature vector from the noise distribution and generate corresponding interactive behavior instructions; in the inference stage, the input is the encoded feature vector, and the model starts with noise and gradually performs denoising operations to finally generate interactive behavior instructions.

[0103] Alternatively, the flow-matching method can be used. Inputting noisy actions, the model is trained using a flow-matching loss function, learning a "flow" or deterministic path directly from random noise to the expert action sequence. During inference, the model starts from a random point and integrates along the learned vector field, efficiently and deterministically generating a smooth, high-frequency continuous action trajectory. Compared to traditional diffusion models, the flow-matching method eliminates the need for multiple noise addition and removal steps, directly generating action sequences from random noise, thus reducing computational load during inference and improving generation efficiency.

[0104] In this embodiment, the ViT model effectively extracts features of interaction scene information and pose information, captures global relationships, and provides high-quality feature representations for subsequent processing. The Transformer temporal information encoding model can handle dynamic changes in interaction behavior, considering the temporal relationships between feature vectors, making the generated interaction commands more consistent with the actual interaction process. The Diffusion policy model can learn complex mapping relationships and generate diverse interaction commands. Furthermore, since each model can be independently optimized and improved, and can be replaced or extended according to specific needs, it has good scalability. In one embodiment, such as... Figure 6 As shown, the method also includes:

[0105] Step 602: Determine the processor type of the bionic robot.

[0106] The processor type refers to the specific model and architecture of the central processing unit (CPU), graphics processing unit (GPU), or other dedicated processing chip used in the bionic robot. Different processors have different computing capabilities, instruction sets, memory bandwidth, and other characteristics, which affect the model's running efficiency and performance on the processor.

[0107] Optionally, you can consult the product manual or technical documents of the bionic robot, which usually clearly indicate the processor model and related information. If you cannot obtain the document information, you can use the system information tool of the operating system to view the basic information of the processor, or you can use a dedicated chip identification tool. These tools can more accurately identify the detailed information of the processor, including its architecture, number of cores, frequency, etc.

[0108] Step 604: Convert the format of the trained behavior generation model based on the processor type to obtain the behavior generation model.

[0109] Format conversion is crucial for enabling trained behavior generation models to run efficiently on specific processors. Different processors have varying requirements for model format and data representation. For instance, some processors may support specific computing libraries or instruction sets, necessitating model conversion to a compatible format to fully utilize the processor's performance. Furthermore, format conversion can optimize model structure, reduce memory usage and computational load, and improve inference speed.

[0110] Optionally, since many processor-compatible inference frameworks support the ONNX (Open Neural Network Exchange) format as an intermediate conversion bridge, you can first convert the trained behavior generation model using a PyTorch-trained model to ONNX format (intermediate format), and then perform the format conversion according to the specific processor type. Taking a CPU as the target processor, you can first select a suitable inference framework, such as MNN, and then use the MNNConvert tool to convert the ONNX model to MNN format. You can specify parameters such as the input ONNX file and the output MNN file path in the command line. Next, perform quantization, such as quantizing the model to 8 bits. This reduces the model's memory usage and computational load, improving its inference speed on the CPU. Finally, prepare a set of test data similar to a real-world application scenario, load the converted model using the inference framework corresponding to the target processor, and perform inference using the test data. Compare the inference results of the converted model with those of the original model to ensure that the conversion process did not introduce significant errors. If the results do not meet expectations, check the parameter settings during the conversion process or perform the conversion again.

[0111] In this embodiment, by converting the behavior generation model into a format suitable for the processor type, the processor's computing power can be fully utilized, improving the model's inference speed and processing efficiency. In addition, format conversion can optimize the model, reducing its memory usage and computational load.

[0112] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0113] Based on the same inventive concept, this application also provides a bionic robot interactive behavior control device for implementing the aforementioned bionic robot interactive behavior control method. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the bionic robot interactive behavior control device provided below can be found in the limitations of the bionic robot interactive behavior control method described above, and will not be repeated here.

[0114] In one embodiment, such as Figure 7 As shown, a biomimetic robot interactive behavior control device is provided, including: a data acquisition module 702, an instruction generation module 704, and a behavior control module 706, wherein:

[0115] The data acquisition module 702 is used to acquire interaction scene information and the pose information of the bionic robot in the scenario where the bionic robot interacts with the interactive object.

[0116] The instruction generation module 704 is used to obtain interactive behavior instructions based on interactive scene information, pose information, and a preset behavior generation model. The behavior generation model is obtained by training an initial behavior generation model based on sample interactive scene information, sample pose information of the bionic robot, and gold standard interactive instructions. The gold standard interactive instructions are generated from the interactive behavior data of the bionic object from the perspective of the reconstructed bionic object based on sample video data of the interaction between the bionic robot's bionic object and the interactive object.

[0117] The behavior control module 706 is used to generate corresponding interactive behaviors based on interactive behavior instructions.

[0118] In one embodiment, the device is further configured to: construct a three-dimensional scene corresponding to the sample video data based on a preset three-dimensional scene construction algorithm; in the three-dimensional scene, reconstruct the interaction behavior and interaction trajectory of the bionic object and the interactive object in the sample video data using a preset depth estimation algorithm; simulate and render the reconstructed interaction behavior and the reconstructed interaction trajectory to generate interaction behavior data of the bionic object from the perspective of the bionic object; and process the interaction behavior data using a key point detection algorithm to obtain the gold standard interaction command.

[0119] In one embodiment, the device is further configured to: determine the joints of the bionic object using a key point detection algorithm; and map the interactive behavior data to the joints of the bionic object using a motion redirection algorithm to obtain the gold standard interactive instructions.

[0120] In one embodiment, the device is further configured to: construct a virtual bionic object of the same size as the bionic object in a virtual interactive scene; adjust the position and posture of the virtual bionic object at each time point according to the reconstructed interactive behavior and the reconstructed interactive trajectory, and generate interactive behavior data of the bionic object from the perspective of the bionic object.

[0121] In one embodiment, the device is further configured to: process the sample interaction scene information and sample pose information using a ViT model to obtain corresponding feature vectors; use a Transformer temporal information encoding model to determine the temporal relationship between feature vectors to obtain encoded feature vectors corresponding to the feature vectors; and use a Diffusion policy model to learn the encoded feature vectors to obtain interaction behavior instructions.

[0122] In one embodiment, the device is further configured to: determine the processor type of the bionic robot; and perform format conversion on the trained behavior generation model based on the processor type to obtain the behavior generation model.

[0123] Each module in the aforementioned biomimetic robot interactive behavior control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0124] In one embodiment, a computer device is provided, which may be a biomimetic robot, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a biomimetic robot interactive behavior control method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0125] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0127] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0128] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0129] It should be noted that the interactive object information (including but not limited to interactive object device information, interactive object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the interactive object or fully authorized by all parties.

[0130] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for controlling interaction behavior of a biomimetic robot, characterized by, The method comprises: In the scenario of interaction between a bionic robot and an interactive object, interaction scene information and pose information of the bionic robot are acquired; According to the interaction scene information, the pose information, and a preset behavior generation model, an interaction behavior instruction is obtained; the behavior generation model is obtained by training an initial behavior generation model according to sample interaction scene information, sample pose information of the bionic robot, and a gold standard interaction instruction; the gold standard interaction instruction is generated by the following steps: based on a preset three-dimensional scene construction algorithm, a three-dimensional scene corresponding to sample video data of an interactive object and a bionic object of the bionic robot is constructed; in the three-dimensional scene, a preset depth estimation algorithm is used to reconstruct an interaction behavior and an interaction trajectory of the bionic object and the interactive object in the sample video data; a virtual bionic object with the same size as the bionic object is constructed in a virtual interaction scene; according to the reconstructed interaction behavior and the reconstructed interaction trajectory, the position and the pose of the virtual bionic object at each time point are adjusted to generate interaction behavior data of the bionic object from the perspective of the bionic object; the interaction behavior data is processed by using a key point detection algorithm to obtain the gold standard interaction instruction; Based on the interaction behavior instruction, a corresponding interaction behavior is generated.

2. The method of claim 1, wherein, The preset depth estimation algorithm comprises a target detection algorithm and a tracking algorithm; The preset depth estimation algorithm comprises a target detection algorithm and a tracking algorithm; The preset depth estimation algorithm comprises a target detection algorithm and a tracking algorithm; The target detection algorithm is used to detect the position, orientation, and bounding box of the bionic object and the interactive object in each frame of the sample video data; The tracking algorithm is used to track the position, orientation, and bounding box of the bionic object and the interactive object between consecutive frames; 3. The method of claim 1, wherein, The position of the bionic object and the interactive object in three-dimensional space is connected in time sequence by combining the depth information and the target position of each frame of the sample video data to generate the interaction behavior and the interaction trajectory on 4D time sequence. The key point detection algorithm is used to determine the joint points of the bionic object; The interaction behavior data is mapped to the joint points of the bionic object by using a motion reorientation algorithm to obtain the gold standard interaction instruction.

4. The method of claim 3, wherein, The key point detection algorithm is used to determine the joint points of the bionic object; The interaction behavior data is mapped to the joint points of the bionic object by using a motion reorientation algorithm to obtain the gold standard interaction instruction. The key point detection algorithm is used to determine the joint points of the bionic object; The three-dimensional coordinates of the detected limb joint points of the bionic object are mapped to the skeletal structure of the bionic robot, and the rotation and translation parameters of the corresponding joints are calculated according to the similarity between the skeletal structure of the bionic object and the skeletal structure of the bionic robot to obtain the gold standard interaction instruction.

5. The method of claim 1, wherein, The behavior generation model comprises a ViT model, a Transformer time sequence information encoding model, and a Diffusion policy model; and the interaction behavior instruction is obtained according to the interaction scene information, the pose information, and the preset behavior generation model. The sample interaction scene information and the sample pose information are processed by using the ViT model to obtain corresponding feature vectors. The time sequence relationship between the feature vectors is determined by using the Transformer time sequence information encoding model to obtain encoded feature vectors corresponding to the feature vectors. The encoded feature vectors are sampled and denoised by using the Diffusion policy model to obtain the interaction behavior instruction.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining the processor type of the bionic robot; format-converting the trained behavior generation model based on the processor type to obtain the behavior generation model.

7. A bionic robot interactive behavior control device, characterized in that, The apparatus comprises: a data acquisition module configured to acquire interaction scene information and pose information of a bionic robot in a scenario in which the bionic robot interacts with an interactive object; an instruction generation module configured to obtain an interaction behavior instruction according to the interaction scene information, the pose information, and a preset behavior generation model; the behavior generation model is obtained by training an initial behavior generation model based on sample interaction scene information, sample pose information of the bionic robot, and a gold standard interaction instruction; the gold standard interaction instruction is generated by the following steps: constructing a three-dimensional scene corresponding to sample video data of a bionic object of the bionic robot and an interactive object based on a preset three-dimensional scene construction algorithm; reconstructing an interaction behavior and an interaction trajectory of the bionic object and the interactive object in the sample video data in the three-dimensional scene by using a preset depth estimation algorithm; constructing a virtual bionic object having the same size as the bionic object in a virtual interaction scene; adjusting a position and a pose of the virtual bionic object at each time point based on the reconstructed interaction behavior and the reconstructed interaction trajectory to generate interaction behavior data of the bionic object from the perspective of the bionic object; and processing the interaction behavior data by using a key point detection algorithm to obtain the gold standard interaction instruction; a behavior control module configured to generate a corresponding interaction behavior based on the interaction behavior instruction.

8. A biomimetic robot comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images