Robot end effector trajectory generation method, data generation method and device

Abstract

The application discloses a robot end effector trajectory generation method, a data generation method and device. The trajectory generation method is based on a first transformation matrix, a second transformation matrix and an internal parameter matrix of an image acquisition device. Through three coordinate transformations, a first coordinate transformation is from a base coordinate system to a world coordinate system, a second coordinate transformation is from the world coordinate system to the image acquisition device coordinate system, and a third coordinate transformation is from the image acquisition device coordinate system to an image pixel plane. A three-dimensional space position matrix of the end effector in the base coordinate system is mapped to the image pixel plane of a video image to generate a two-dimensional trajectory. Through the first coordinate transformation and the second coordinate transformation, the three-dimensional space position matrix of the end effector in the base coordinate system and the image acquisition device parameters are uniformly transformed to the world coordinate system, so that the two-dimensional trajectory can correctly reflect the visual convergence effect when the robot moves, the accuracy of trajectory drawing is improved, and the two-dimensional trajectory can be automatically generated, thereby saving time and effort.

Description

Technical Field

[0001] This invention relates to the field of robot kinematics technology, and more specifically, to a method, data generation method, and apparatus for generating trajectory of a robot end effector. Background Technology

[0002] With the rapid development of artificial intelligence technology, embodied intelligence has become an important research direction in the field of robotics. Visual language models play a key role in embodied intelligence systems, and their training relies on large-scale image-text alignment data. For humanoid robot manipulation tasks, a key requirement is to enable the model to understand "where the robot's end effector is moving from the current perspective," that is, the two-dimensional projection position of the end effector's two-dimensional trajectory on the image acquisition device screen.

[0003] Currently, the existing methods for projecting the two-dimensional trajectory of an end effector onto the two-dimensional image acquisition device screen have the following main shortcomings: First, there is a lack of large-scale automated data production methods: Traditional methods primarily rely on manual annotation or simulation environment rendering. Manual annotation is time-consuming, labor-intensive, and costly, and it is difficult to guarantee consistency. While simulation rendering can automatically generate data, it suffers from significant discrepancies between the virtual and real worlds, resulting in poor accuracy that is difficult to directly transfer to real systems. More importantly, current technologies cannot automatically extract the two-dimensional trajectory information of the end effector from the massive amounts of collected robot teleoperation data, thus failing to fully utilize real operation data resources.

[0004] Second, ignore the influence of chassis motion on trajectory projection: For robots equipped with mobile chassis, current forward kinematics calculations are typically performed in the robot arm base coordinate system. Since the image acquisition device is fixed to the robot, the relative relationship between the end effector and the image acquisition device in the robot arm base coordinate system remains unchanged. When the robot chassis moves, if trajectory projection is performed solely based on the robot arm base coordinate system, the drawn trajectory will rigidly move with the robot as a whole, rather than contracting or expanding towards the target point. This is severely inconsistent with actual visual observation, resulting in poor trajectory accuracy and misleading subsequent model learning.

[0005] In summary, the existing method of projecting the two-dimensional trajectory of an end effector onto the two-dimensional image acquisition device screen suffers from time-consuming, labor-intensive, and inaccurate problems. Summary of the Invention

[0006] This invention provides a method, data generation method, and apparatus for generating the trajectory of a robot end effector, which can save time and effort and improve accuracy. The specific technical solution is as follows.

[0007] In a first aspect, the present invention provides a method for generating the trajectory of a robot end effector, comprising: Acquire teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device; Based on the joint state data, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device are calculated. Based on the motion data, a second transformation matrix from the world coordinate system to the base coordinate system is calculated; Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system is projected onto the video image to generate a two-dimensional trajectory.

[0008] Optionally, the motion data includes wheel speed data and steering angle data, and the step of calculating the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data includes: Based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data, the body speed is obtained through inverse kinematics calculation. Based on the machine velocity, the second transformation matrix from the world coordinate system to the base coordinate system is calculated recursively through numerical integration.

[0009] Optionally, the step of projecting the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device includes: The first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device are combined to obtain the normalized projection matrix; The homogeneous pixel coordinates are obtained by multiplying the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix. Perform perspective division on the homogeneous pixel coordinates to obtain normalized pixel coordinates; Draw the two-dimensional trajectory points corresponding to the normalized pixel coordinates on the video image.

[0010] Secondly, the present invention provides a data generation method, comprising: Acquire teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device; Based on the joint state data, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device are calculated. Based on the motion data, a second transformation matrix from the world coordinate system to the base coordinate system is calculated; Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm is projected onto the video image to generate a two-dimensional trajectory. Multiple sampling frames are obtained by sampling at the boundary frames of atomic tasks, wherein the teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks have changed; Based on the two-dimensional trajectory of each sampling frame, corresponding labeled sample data for model training are generated according to the type of visual question answering task.

[0011] Optionally, when the visual question answering task is a trajectory coordinate prediction task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame according to the type of visual question answering task includes: For each sampled frame, a first visual question answering task is constructed, wherein the input of the first visual question answering task includes the video image of the two-dimensional trajectory of the sampled frame and the question text generated based on the preset question template, which includes the first high-level task instruction and the current atomic task corresponding to the sampled frame; Determine the future two-dimensional trajectory of the end effector corresponding to the sampling frame, and use it as the answer to the first visual question answering task of the sampling frame; Based on the input and answer of the first visual question-answering task in each sample frame, the first labeled sample data for model training is generated.

[0012] Optionally, when the visual question answering task is a trajectory overlay multiple-choice task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectories of each sampling frame according to the type of visual question answering task includes: For each sampled frame, the three-dimensional spatial position matrix of the end effector of the future frame is projected onto the video image of the sampled frame through the intrinsic parameter matrix of the image acquisition device of the sampled frame by cross-frame reprojection; The two-dimensional trajectory points obtained by projection are drawn onto the video image of the sampled frame to generate a trajectory overlay map; Construct a second visual question answering task, wherein the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampling frame and the second high-level task instruction; Based on the atomic task label corresponding to the teleoperation data in which the sampled frame is located, the first correct atomic task is determined and used as the first correct answer option; Based on the second high-level task instruction, the first correct atomic task, and the list of other atomic tasks under the second high-level task, generate multiple first interference options that are semantically similar but have different actions; The first correct answer option and multiple first interference options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the first correct answer option is used as the answer to the second visual question answering task of the sampled frame. Based on the input, selection options, and answers of the second visual question answering task in each sample frame, second labeled sample data for model training is generated.

[0013] Optionally, when the visual question answering task is an atomic task planning multiple-choice question task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame according to the type of visual question answering task includes: For each sampled frame, a third visual question answering task is constructed, wherein the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame and the third high-level task instruction; By analyzing motion behavior and semantic annotation, the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame is determined and used as the second correct answer option; Based on the third high-level task instruction, the second correct atomic task, and the list of other atomic tasks under the third high-level task, generate multiple second interference options that are semantically similar but have different actions; The second correct answer option and multiple second distractor options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the second correct answer option is used as the answer to the third visual question answering task of the sampled frame. Based on the input, multiple-choice options, and answers of the third visual question-answering task in each sample frame, third labeled sample data for model training is generated.

[0014] Thirdly, embodiments of the present invention provide a robot end effector trajectory generation device, comprising: The first acquisition module is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The first calculation module is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data. The second calculation module is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; The first trajectory projection module is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device.

[0015] Optionally, the motion data includes wheel speed data and steering angle data, and the second calculation module includes: The body speed calculation submodule is used to obtain the body speed through inverse kinematics calculation based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data. The second transformation matrix calculation submodule is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system by numerical integration based on the body velocity.

[0016] Optionally, the first trajectory projection module includes: The merging submodule is used to merge the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device to obtain a normalized projection matrix; The multiplication submodule is used to multiply the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix to obtain homogeneous pixel coordinates; The normalized pixel coordinate determination submodule is used to perform perspective division on the homogeneous pixel coordinates to obtain normalized pixel coordinates; The drawing submodule is used to draw the two-dimensional trajectory points corresponding to the normalized pixel coordinates on the video image.

[0017] Fourthly, embodiments of the present invention provide a data generation apparatus, comprising: The second acquisition module is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The third calculation module is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data. The fourth calculation module is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; The second trajectory projection module is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device. The sampling module is used to perform sampling at the boundary frames of atomic tasks to obtain multiple sampling frames, wherein the teleoperation data includes multiple atomic tasks, and the boundary frames are frames where atomic tasks have changed; The labeled sample data generation module is used to generate corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame, according to the type of visual question answering task.

[0018] Optionally, when the visual question answering task is a trajectory coordinate prediction task, the labeled sample data generation module includes: The first construction submodule is used to construct a first visual question answering task for each sampled frame. The input of the first visual question answering task includes a video image of the two-dimensional trajectory of the sampled frame and a question text generated based on a preset question template, which includes instructions for a first high-level task and the current atomic task corresponding to the sampled frame. The first answer determination submodule is used to determine the future two-dimensional trajectory of the end effector corresponding to the sampling frame, and to serve as the answer to the first visual question answering task of the sampling frame; The first generation submodule is used to generate the first labeled sample data for model training based on the input and answer of the first visual question answering task in each sampled frame.

[0019] Optionally, when the visual question answering task is a trajectory overlay multiple-choice task, the labeled sample data generation module includes: The projection submodule is used to project the three-dimensional spatial position matrix of the end effector of the future frame onto the video image of the sample frame through cross-frame reprojection for each sample frame. The trajectory overlay generation submodule is used to draw the two-dimensional trajectory points obtained by projection onto the video image of the sampled frame to generate a trajectory overlay. The second construction submodule is used to construct the second visual question answering task, wherein the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampling frame and the second high-level task instruction; The first correct answer option determination submodule is used to determine the first correct atomic task based on the atomic task label in the teleoperation data where the sampled frame is located, and use it as the first correct answer option; The first interference option generation submodule is used to generate multiple first interference options with similar semantics but different actions based on the second high-level task instruction, the first correct atomic task and the list of other atomic tasks under the second high-level task. The second answer determination submodule is used to randomly arrange the first correct answer option and multiple first interference options to form multiple choice options, and assign corresponding option identifiers, and find the option identifier corresponding to the first correct answer option as the answer to the second visual question answering task of the sampled frame; The second generation submodule is used to generate second labeled sample data for model training based on the input, selection options and answers of the second visual question answering task based on each sampled frame.

[0020] Optionally, when the visual question answering task is an atomic task planning multiple-choice question task, the labeled sample data generation module includes: The third construction submodule is used to construct a third visual question answering task for each sampled frame, wherein the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame and the third high-level task instruction; The second correct answer option determination submodule is used to determine the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame through motion behavior analysis and semantic annotation, and to serve as the second correct answer option; The second interference option generation submodule is used to generate multiple second interference options with similar semantics but different actions based on the third high-level task instruction, the second correct atomic task and the list of other atomic tasks under the third high-level task. The third answer determination submodule is used to randomly arrange the second correct answer option and multiple second interference options to form multiple choice options, and assign corresponding option identifiers. The option identifier corresponding to the second correct answer option is found as the answer to the third visual question answering task of the sample frame. The third generation submodule is used to generate third labeled sample data for model training based on the input, selection options and answers of the third visual question answering task based on each sampled frame.

[0021] As described above, this invention provides a method for generating a robot end effector trajectory. The method includes: acquiring teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by an image acquisition device; calculating a three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and a first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data; calculating a second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; and projecting the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video images to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device. Therefore, based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, through three coordinate transformations—the first from the base coordinate system to the world coordinate system, the second from the world coordinate system to the image acquisition device coordinate system, and the third from the image acquisition device coordinate system to the image pixel plane—the three-dimensional spatial position matrix of the end effector in the base coordinate system is mapped to the image pixel plane of the video image to generate a two-dimensional trajectory. By transforming the three-dimensional spatial position matrix of the end effector in the base coordinate system and the parameters of the image acquisition device into the world coordinate system through the first and second coordinate transformations, the two-dimensional trajectory obtained by the third coordinate transformation can correctly reflect the visual convergence effect when the robot moves, rather than moving rigidly with the robot as a whole, thus improving the accuracy of trajectory drawing. Furthermore, it can also realize the automatic generation of two-dimensional trajectories based on teleoperation data, saving time and effort.

[0022] The innovative aspects of this invention include: 1. Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, three coordinate transformations are performed: the first transformation from the base coordinate system to the world coordinate system, the second transformation from the world coordinate system to the image acquisition device coordinate system, and the third transformation from the image acquisition device coordinate system to the image pixel plane. This maps the three-dimensional spatial position matrix of the end effector in the base coordinate system to the image pixel plane of the video image, generating a two-dimensional trajectory. The first and second coordinate transformations unify the three-dimensional spatial position matrix of the end effector in the base coordinate system and the parameters of the image acquisition device to the world coordinate system. This ensures that the two-dimensional trajectory obtained by the third coordinate transformation can accurately reflect the visual convergence effect when the robot moves, rather than rigidly moving with the robot as a whole, thus improving the accuracy of trajectory drawing. Furthermore, it can automatically generate two-dimensional trajectories based on teleoperation data, saving time and effort.

[0023] 2. Based on joint angle data and the robot's geometric model, the forward kinematics function is called to perform kinematic calculations, obtaining the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robot arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device. This achieves accurate representation of the end effector in the base coordinate system of the robot arm, overcomes the perception error caused by changes in head posture due to the fixed constant extrinsic parameters of the image acquisition device, and realizes dynamic updating of the extrinsic parameters of the image acquisition device with the robot's motion state.

[0024] 3. Based on the kinematic model, wheel speed data, and steering angle data of the mobile chassis, the body velocity is obtained through inverse kinematics calculation. Then, based on the body velocity, the second transformation matrix from the world coordinate system to the base coordinate system is calculated recursively through numerical integration. This converts the local motion increment into the global absolute pose, realizing the robot's real-time localization, unified alignment of sensor data, and high-precision motion control. This provides a basic spatial reference framework for autonomous navigation, multi-sensor fusion, and task execution.

[0025] 4. By merging the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, a normalized projection matrix is ​​obtained. The three-dimensional spatial position matrix of the end effector in the world coordinate system is multiplied by the normalized projection matrix to obtain homogeneous pixel coordinates. Perspective division is performed on the homogeneous pixel coordinates to obtain normalized pixel coordinates. Then, the two-dimensional trajectory points corresponding to the normalized pixel coordinates are drawn on the video image to realize the projection of the two-dimensional trajectory onto the video image. This two-dimensional trajectory takes into account the influence of the viewpoint change caused by the rotation of the image acquisition device to avoid trajectory drawing distortion.

[0026] 5. For each sampled frame, a first visual question-answering task is constructed. The input to the first visual question-answering task includes the video image containing the 2D trajectory of the sampled frame and question text generated based on a preset question template, containing the first high-level task instruction and the current atomic task corresponding to the sampled frame. This determines the future 2D trajectory of the end effector corresponding to the sampled frame and serves as the answer to the first visual question-answering task for that sampled frame. Based on the input and answer of the first visual question-answering task for each sampled frame, first labeled sample data for model training is generated. This achieves the goal of automatically generating first labeled sample data for model training corresponding to the trajectory coordinate prediction task based on the video image containing the 2D trajectory generated from the robot's teleoperation data, eliminating the need for manual annotation and significantly reducing the cost of dataset construction. Simultaneously, through the trajectory coordinate prediction task, the model is trained to understand the reasonable operation trajectory in the current scene, thereby enabling it to predict the 2D trajectory based on new scenes.

[0027] 6. By projecting the 3D spatial position matrix of the future end effector onto the current video image through cross-frame reprojection, a trajectory overlay map is obtained as a visual cue. Semantically similar distracting options are generated to ensure that the multiple-choice questions have reasonable difficulty, avoiding the model's learning through simple pattern matching. The second high-level task instructions are deeply integrated with each atomic task. Through atomic task multiple-choice questions, the model is trained to identify the currently executed atomic task, enhancing its task state perception capability. Finally, based on the input of the second visual question-and-answer task, the multiple-choice options, and the answers of each sampled frame, second labeled sample data for model training is generated. This achieves the goal of automatically generating second labeled sample data for model training corresponding to the trajectory overlay multiple-choice task based on the trajectory overlay map generated by cross-frame reprojection of robot teleoperation data. Simultaneously, through the trajectory overlay multiple-choice task, the model is trained to infer motion intent from trajectory shape and select the correct atomic task from multiple options.

[0028] 7. Based on the third high-level task instruction, the second correct atomic task, and the list of other atomic tasks under the third high-level task, generate semantically similar interference options to ensure that the multiple-choice questions have reasonable difficulty and avoid the model learning through simple pattern matching. Deeply integrate the third high-level task instruction with each atomic task. Through atomic task multiple-choice questions, train the model to identify the currently executed atomic task and enhance task state perception capabilities. Finally, based on the input, multiple-choice options, and answers of the third visual question-answering task in each sample frame, generate third-labeled sample data for model training. This achieves the goal of automatically generating third-labeled sample data for model training corresponding to the atomic task planning multiple-choice task based on the two-dimensional trajectory generated by the robot's teleoperation data. Simultaneously, through the atomic task planning multiple-choice task, train the model to infer the correct trajectory semantics from the two-dimensional trajectory and select the correct atomic task from multiple options.

[0029] 8. After generating the 2D trajectory, sampling is performed at the boundary frames of the atomic tasks to obtain multiple sample frames. The teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks change. Then, based on the 2D trajectory of each sample frame, corresponding labeled sample data for model training is generated according to the type of visual question answering task. This achieves the goal of transforming trajectory data into labeled sample data for model training corresponding to different types of visual question answering tasks, thereby improving the model's ability to understand the robot's operational trajectory.

[0030] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0032] Figure 1 A schematic flowchart of a robot end effector trajectory generation method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a data generation method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a robot end effector trajectory generation device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a data generation device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0035] This invention discloses a method, data generation method, and apparatus for generating the trajectory of a robot end effector, which can save time and effort, improve accuracy, and avoid distortion. The embodiments of this invention are described in detail below.

[0036] Example 1 Figure 1 This is a flowchart illustrating a robot end effector trajectory generation method provided in an embodiment of the present invention. The method is applied to electronic devices. Specifically, the method includes the following steps: S110: Acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device.

[0037] To project the three-dimensional spatial position of the robot's end effector, it is necessary to acquire the robot's teleoperation data, which includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device.

[0038] Teleoperation data consists of robot motion data, comprising N task execution segments. Each task execution segment contains all relevant data from the start to the end of a complete task execution, specifically including multiple data frames. The teleoperation data format can be Parquet. The mobile chassis can be a three-wheeled omnidirectional chassis, which is an omnidirectional chassis with three steerable drive wheels. The image acquisition device can be a camera, which can be fixed to the robot's base, the end effector of the robotic arm, or the mobile chassis.

[0039] Since the robot has a left arm and a right arm, the joint state data includes the joint state data of the left and right arms. Specifically, the joint state data can be joint angle data.

[0040] S120: Based on the joint state data, calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system.

[0041] After acquiring the teleoperation data, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system are calculated based on the joint state data. The first transformation matrix represents the transformation from the base coordinate system to the image acquisition device coordinate system.

[0042] The joint state data is joint angle data, and step S120 includes: Based on joint angle data and the robot's geometric model, the forward kinematics function is called to perform kinematic calculations, obtaining the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system.

[0043] Specifically, joint angle data can include torso joint angle data and arm joint angle data. Based on the joint angle data and the robot's geometric model, kinematic calculations are performed using forward kinematics functions to obtain the three-dimensional spatial position matrix of the end effector in the robot arm's base coordinate system and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system, which can include: The torso joint angle data and the arm joint angle data are concatenated to obtain a concatenated vector. Based on the robot's geometric model and positive kinematics function, the third transformation matrix from the robot arm's base coordinate system to the gripper coordinate system is calculated on the concatenated vector. Multiplying the third transformation matrix and the preset offset transformation matrix from the gripper to the end effector yields the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm; The fourth transformation matrix from the base coordinate system to the torso joint coordinate system of the robotic arm is obtained by calculating the torso joint angle data based on the robot's geometric model and positive kinematic functions. Multiply the fourth transformation matrix by the pre-calibrated fixed transformation matrix from the torso joint coordinate system to the image acquisition device coordinate system to obtain the first transformation matrix from the base coordinate system of the robotic arm to the image acquisition device coordinate system.

[0044] In this embodiment of the invention, the trunk and arm degrees of freedom can be automatically inferred by querying the number of joint parameter names in the kinematic chain, thereby obtaining trunk joint angle data and arm joint angle data. It is compatible with both R1 Lite and R1 Pro robots without manual configuration.

[0045] The third transformation matrix from the base coordinate system to the gripper coordinate system of the robotic arm, calculated from the spliced ​​vectors based on the robot's geometric model and positive kinematic functions, can be: The static structural information of the robot is obtained by parsing the robot's geometric model file; Substituting the static structural information and the spliced ​​vector into the forward kinematics function, the third transformation matrix from the base coordinate system to the gripper coordinate system of the robotic arm is calculated.

[0046] The robot's geometric model file defines "what the robot looks like," including its static structural information. This static structural information can include link information, joint information, position and orientation offsets between adjacent links, and the direction and range of motion of the joints. The forward kinematics function uses this static structural information to calculate "where the hand is," that is, the pose of the end effector. Without a geometric model file, the forward kinematics function doesn't know the length of each link and the direction of the joints; without a forward kinematics function, the geometric model file is merely a static description file. Specifically, the geometric model file can be a URDF (Unified Robot Description Format) file.

[0047] The gripper is the actuator at the end of a robotic arm, responsible for grasping objects. The end effector is a manually defined control point on the gripper, typically located at the center of the gripper's fingertips, the gripping center, or the tool tip. Once the gripper is installed and calibrated, the offset transformation matrix between the gripper and the end effector is fixed and will not change.

[0048] Therefore, after obtaining the third transformation matrix, the third transformation matrix can be multiplied by the preset offset transformation matrix from the gripper to the end effector to obtain the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robot arm. This three-dimensional spatial position matrix represents the position of the end effector relative to the origin of the base coordinate system of the robot arm.

[0049] Similarly, based on the robot's geometric model and positive kinematics functions, the fourth transformation matrix from the robot arm's base coordinate system to the torso joint coordinate system is calculated from the trunk joint angle data.

[0050] Then, the fourth transformation matrix is ​​multiplied by the pre-calibrated fixed transformation matrix from the torso joint coordinate system to the image acquisition device coordinate system to obtain the first transformation matrix from the base coordinate system of the robotic arm to the image acquisition device coordinate system.

[0051] Therefore, based on joint angle data and the robot's geometric model, the forward kinematics function is called to perform kinematic calculations, obtaining the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robot arm and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system. This achieves accurate representation of the end effector in the base coordinate system of the robot arm, overcomes the perception error caused by changes in head posture due to the fixed constant extrinsic parameters of the image acquisition device, and realizes dynamic updating of the extrinsic parameters of the image acquisition device with the robot's motion state.

[0052] In this embodiment of the invention, the calculation logic for the end effector of the left and right arms is the same; it is only necessary to input the corresponding parameter values, which will not be elaborated here.

[0053] S130: The second transformation matrix from the world coordinate system to the base coordinate system is calculated based on the motion data.

[0054] Specifically, the second transformation matrix from the world coordinate system to the base coordinate system is calculated based on the motion data.

[0055] The motion data includes wheel speed data and steering angle data. Step S130 includes: Based on the kinematic model of the mobile chassis, wheel speed data, and steering angle data, the body speed is obtained through inverse kinematics calculation. Based on the body velocity, the second transformation matrix from the world coordinate system to the base coordinate system is obtained by numerical integration recursively.

[0056] Specifically, based on the kinematic model of the mobile chassis, wheel speed data, and steering angle data, the body velocity is obtained through inverse kinematics calculation, including: A forward kinematics matrix is ​​constructed based on the kinematic model of the mobile chassis, and an inverse kinematics matrix is ​​calculated based on the forward kinematics matrix. A wheel speed vector is constructed based on wheel speed data and steering angle data; Multiplying the wheel velocity vector by the inverse kinematics matrix yields the body velocity.

[0057] Taking a three-wheeled omnidirectional chassis as an example, a 6×3 positive kinematic matrix F is constructed based on the kinematic model of the chassis. This matrix is ​​used to map the chassis velocity vector [vx, vy, ωz] to the 6-dimensional wheel velocity vector [v1x, v1y, v2x, v2y, v3x, v3y]. The above mapping relationship satisfies: F · [vx, vy,ωz] T = [v1x, v1y, v2x, v2y, v3x, v3y] T Where F is the positive kinematics matrix, vx is the linear velocity component along the X-axis, vy is the linear velocity component along the Y-axis, ωz is the angular velocity component about the Z-axis, and v1x, v1y, v2x, v2y, v3x, and v3y are the velocity components of the three drive wheels in the X and Y directions, respectively. The three drive wheels are the front wheel, left wheel, and right wheel.

[0058] The positive kinematic matrix F is in the following form: F = [[1, 0, -b ], [0, 1, a], [1, 0, b], [0, 1, a+ε], [1, 0, 0 ], [0, 1, -r ]] Where r is the chassis radius, a is the longitudinal offset of the front wheel, b is the lateral distance between the left and right wheels, and ε is a numerical stability term used to avoid matrix singularity.

[0059] Taking the R1 Pro robot as an example: chassis radius r=0.32703m, front wheel longitudinal offset a=0.16897m, left and right wheel lateral distance b=0.280m, ε=0.01.

[0060] The inverse kinematics matrix, calculated from the forward kinematics matrix, can be: A 3×6 inverse kinematics matrix W is constructed and solved using the Tikhonov regularized pseudoinverse method to improve numerical stability and suppress the error amplification effect caused by ill-conditioned matrices. The formula for calculating the inverse kinematics matrix W is as follows: W = (F T F +λI) -1 F T , λ = 10 -6 Where W is the inverse kinematics matrix, F is the forward kinematics matrix, and λ is the regularization coefficient, which can be 10. -6 I is the identity matrix. It should be noted that W = (F T F +λI) -1 F T This is a numerical algorithm formula, not a physical formula. In the numerical implementation, all physical quantities have been substituted into the numerical values; λI is only used as a numerical regularization term to improve the condition number of the matrix and is not required to be equal to F. T F maintains consistency in physical dimensions. This is standard practice in engineering calculations.

[0061] Based on wheel speed data and steering angle data, the wheel speed vector can be constructed as follows: Based on the wheel speed and steering angle of each wheel of the mobile chassis, the motion state of each wheel is converted into velocity components in the Cartesian coordinate system, and a wheel speed vector is generated based on each velocity component.

[0062] After obtaining the wheel velocity vector, multiply the wheel velocity vector by the inverse kinematics matrix to obtain the body velocity.

[0063] Among them, based on the body velocity, the second transformation matrix from the world coordinate system to the base coordinate system is calculated recursively through numerical integration, including: Based on the SE(2) arc motion formula, the body velocity is converted into the corresponding displacement increment and rotation increment; Construct the pose increment transformation matrix based on displacement increment and rotation increment; Using frame 0 as the origin of the world coordinate system, the second transformation matrix from the world coordinate system to the base coordinate system is obtained recursively based on the pose increment transformation matrix of each frame through frame-by-frame chain matrix multiplication.

[0064] The time step dt of each frame is set to be the reciprocal of the frame rate fps, i.e., dt = 1 / fps. The continuous motion process is discretized based on the frame rate. Then, based on the SE(2) arc motion formula, the body velocity is converted into the corresponding displacement increment and rotation increment, and the displacement increment and rotation increment are encoded into the pose increment transformation matrix of each frame.

[0065] Because the simple Euler integral produces accumulated errors at large angular velocities, this embodiment of the invention employs the exact SE(2) arc motion formula, and at small angles (|θ|≤10°). -6 Taylor expansion is used to avoid division-to-zero singularities and ensure numerical stability under all motion conditions.

[0066] Here, taking frame 0 as the origin of the world coordinate system, the second transformation matrix from the world coordinate system to the base coordinate system is obtained recursively based on the pose increment transformation matrix of each frame through frame-by-frame chained matrix multiplication. It can be: With frame 0 as the origin of the world coordinate system, the second transformation matrix from the world coordinate system to the base coordinate system in frame i after frame 0 is obtained by multiplying the second transformation matrix of frame i-1 with the pose increment transformation matrix of frame i.

[0067] Among them, pose recursion solves the problem of where the moving chassis is relative to the world coordinate system.

[0068] Therefore, based on the kinematic model of the mobile chassis, wheel speed data, and steering angle data, the robot velocity is obtained through inverse kinematics calculation. Then, based on the robot velocity, the second transformation matrix from the world coordinate system to the base coordinate system is calculated recursively through numerical integration. This converts the local motion increment into the global absolute pose, realizing the robot's real-time localization, unified alignment of sensor data, and high-precision motion control. This provides a basic spatial reference framework for autonomous navigation, multi-sensor fusion, and task execution.

[0069] When the wheel speed and steering angle data of the mobile chassis are missing in the teleoperation data, it will automatically revert to the base coordinate system processing mode of the robotic arm to ensure backward compatibility with earlier versions of data and consistency of processing results.

[0070] S140: Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system is projected onto the video image to generate a two-dimensional trajectory.

[0071] Because existing forward kinematics calculations are typically performed in the robot arm's base coordinate system, and since the image acquisition device is fixedly mounted on the robot, the relative relationship between the end effector and the image acquisition device in the robot arm's base coordinate system remains unchanged. When the robot's mobile chassis moves, if trajectory projection is performed solely based on the robot arm's base coordinate system, the drawn trajectory will rigidly move with the robot as a whole, rather than contracting or expanding towards the target point. This is severely inconsistent with actual visual observation and cannot reflect the visual impact of the mobile chassis movement on the trajectory. To solve this problem, in this embodiment of the invention, the position of the end effector and the intrinsic parameter matrix of the image acquisition device are both transformed to the world coordinate system.

[0072] Therefore, after obtaining the second transformation matrix, the three-dimensional spatial position matrix of the end effector in the base coordinate system can be projected onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device.

[0073] Specifically, step S140 may include: The first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device are combined to obtain the normalized projection matrix; The homogeneous pixel coordinates are obtained by multiplying the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix. Perform perspective division on the aligned sub-pixel coordinates to obtain normalized pixel coordinates; Draw two-dimensional trajectory points corresponding to normalized pixel coordinates on the video image.

[0074] The normalized projection matrix is ​​obtained by merging the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, including: The intrinsic parameter matrix of the image acquisition device is normalized to obtain the normalized intrinsic parameter matrix; Multiply the first transformation matrix by the inverse of the second transformation matrix to obtain the fifth transformation matrix from the image acquisition device coordinate system to the world coordinate system; Multiply the normalized intrinsic parameter matrix by the fifth transformation matrix to obtain the normalized projection matrix.

[0075] Each frame only needs to store 12 floating-point values ​​of the normalized projection matrix, which are expanded and stored in row-major order of a 3×4 matrix. These 12 floating-point values ​​completely encode the complete transformation chain information of the intrinsic parameter matrix, the first transformation matrix, and the second transformation matrix, thus achieving compact and efficient storage of projection parameters.

[0076] Therefore, by normalizing the process, the projection matrix is ​​decoupled from the image resolution. The same set of parameters can be applied to image processing at different resolutions, adapting to any video resolution. At the same time, the intrinsic parameter matrix, the first transformation matrix, and the second transformation are merged into a single normalized projection matrix, so that the mapping from three-dimensional points to pixels only requires one matrix-vector multiplication, which significantly reduces the computational complexity compared to step-by-step calculation.

[0077] Multiplying the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix yields the homogeneous pixel coordinates, including: The three-dimensional spatial position of the end effector in the world coordinate system is converted into homogeneous coordinates, and the homogeneous coordinates are multiplied by the normalized projection matrix to obtain the homogeneous pixel coordinates.

[0078] Since the normalized projection matrix is ​​3×4, the 3D spatial position needs to be converted to homogeneous coordinates, which is a 4×1 vector [u, v, w, 1]. This conversion is done by adding a homogeneous component 1 to the 3D coordinates, where u represents the X-axis position in the world coordinate system, v represents the Y-axis position, and w represents the Z-axis position. Then, the homogeneous coordinates are multiplied by the normalized projection matrix to obtain the homogeneous pixel coordinates. Finally, perspective division is performed on the homogeneous pixel coordinates, dividing all components of the homogeneous pixel coordinates by the w component to obtain the normalized pixel coordinates.

[0079] Then, draw the two-dimensional trajectory points corresponding to the normalized pixel coordinates on the video image.

[0080] In this process, drawing two-dimensional trajectory points corresponding to normalized pixel coordinates on the video image can use differentiated graphic markers to distinguish the start point, intermediate point, and end point, and use different color codes according to the type of end effector, for example: Starting point: solid circle + white outline Midpoint: Small solid circle End point: Hollow circle Connecting lines: Draw polylines between adjacent points. Color differentiation: green on the left, red on the right.

[0081] Therefore, a normalized projection matrix is ​​obtained by merging the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device. The homogeneous pixel coordinates are obtained by multiplying the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix. The normalized pixel coordinates are obtained by performing perspective division on the homogeneous pixel coordinates. Then, the two-dimensional trajectory points corresponding to the normalized pixel coordinates are drawn on the video image, so as to project the two-dimensional trajectory onto the video image. This two-dimensional trajectory takes into account the influence of the viewpoint change caused by the rotation of the image acquisition device, so as to avoid trajectory drawing distortion.

[0082] In summary, the embodiments of the present invention provide a method for generating the trajectory of a robot end effector. The method includes: acquiring teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by an image acquisition device; calculating a three-dimensional spatial position matrix of the end effector in the base coordinate system of the robot arm and a first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data; calculating a second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; and projecting the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video images to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device. Therefore, based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, through three coordinate transformations—the first from the base coordinate system to the world coordinate system, the second from the world coordinate system to the image acquisition device coordinate system, and the third from the image acquisition device coordinate system to the image pixel plane—the three-dimensional spatial position matrix of the end effector in the base coordinate system is mapped to the image pixel plane of the video image to generate a two-dimensional trajectory. By transforming the three-dimensional spatial position matrix of the end effector in the base coordinate system and the parameters of the image acquisition device into the world coordinate system through the first and second coordinate transformations, the two-dimensional trajectory obtained by the third coordinate transformation can correctly reflect the visual convergence effect when the robot moves, rather than moving rigidly with the robot as a whole, thus improving the accuracy of trajectory drawing. Furthermore, it can also realize the automatic generation of two-dimensional trajectories based on teleoperation data, saving time and effort.

[0083] In one implementation, the robot end effector trajectory generation method provided in this embodiment of the invention further includes: A distributed data parallel processing architecture is adopted, in which the N task execution segments of the teleoperation data are distributed to multiple GPU (Graphics Processing Unit) processes for processing in a round-robin manner; Within each GPU process, multiple subprocess pools are created to concurrently process the allocated task execution segments. Each frame within each task execution segment is processed in parallel using batch tensor operations. Furthermore, during the initialization phase of the subprocess pool, the three-dimensional spatial position matrix, the first transformation matrix, and the second transformation matrix of the end effector corresponding to each task execution segment in the base coordinate system of the robotic arm are cached.

[0084] In this embodiment of the invention, a multi-level parallel processing method is used to process N task execution segments, specifically including three levels: Level 1: DDP (Distributed Data Parallel) cross-GPU parallelism It operates at the task allocation level of task execution segments, adopts a distributed data parallel processing architecture, and distributes N task execution segments to multiple GPU processes for processing in a round-robin manner.

[0085] For example, if there are a total of 4000 task execution segments, distributed among 8 GPU processes, then each GPU process will handle 500 task execution segments.

[0086] Level 2: mp.Pool executes fragments in parallel across tasks It operates on the concurrent processing of multiple task execution segments within a single GPU process. Each GPU process creates multiple child process pools to concurrently process the assigned task execution segments.

[0087] For example, each GPU process has 500 task execution segments distributed to 16 subprocess pools for processing. Each subprocess pool processes approximately 31 task execution segments serially, while the 16 task execution segments work simultaneously, which is concurrent processing.

[0088] Level 3: Parallel processing of batch tensor operations It performs frame-level batch computations within each task execution segment, and uses batch tensor operations to process each frame within each task execution segment in parallel.

[0089] For example, each task execution segment consists of 100 frames, and these 100 frames are computed in parallel.

[0090] During the subprocess pool initialization phase, the 3D spatial position matrix, first transformation matrix, and second transformation matrix of the end effector corresponding to each task execution segment in the robot arm's base coordinate system are cached and generated as global variables. Subsequent task execution segments can reuse these cached global variables, avoiding redundant calculations and reducing overhead.

[0091] Specifically, environment variables ensure that each process can only see its own corresponding GPU, preventing multiple processes from competing for the same GPU and causing resource conflicts. Process states are synchronized via CPU communication, eliminating the need to transfer large amounts of data; only state synchronization, such as start and end signals, is required, thus avoiding the complexity of inter-GPU communication. In a dual-card H20 GPU environment, batch processing of 4000 hours of robot trajectory data can be completed within 3 hours.

[0092] Therefore, a three-layer parallel processing architecture is adopted, consisting of multi-GPU distributed parallelism, in-process subprocess pool cross-task execution fragment parallelism, and frame-level batch tensor computation, combined with process-level caching optimization, to achieve efficient distributed batch processing of robot motion data, thereby improving computational efficiency while reducing overhead.

[0093] In one implementation, the robot end effector trajectory generation method provided in this embodiment of the invention further includes: For each task execution segment included in the teleoperation data, the system checks whether a temporary trajectory directory with a preset name exists in the file system of the task execution segment, whether the feature field list of the metadata information file contains trajectory-related feature fields, and whether the columnar storage file contains a trajectory index column. Based on the judgment results, the processing status of the task execution segment is determined, including completed, data lost, first processing, and rerun / overwrite. When the processing flow for each task execution segment is interrupted and then restarted, breakpoint resumption processing is performed based on the processing status of each task execution segment.

[0094] To ensure the robustness and fault tolerance of large-scale batch processing, this embodiment of the invention designs a four-state breakpoint resume mechanism based on cross-validation of three independent signals.

[0095] The three independent signals are: the signal corresponding to the temporary trajectory directory, the signal corresponding to the trajectory-related feature fields, and the signal corresponding to the trajectory index column. The temporary trajectory directory stores intermediate result files generated during processing; its existence indicates that the processing flow has started or is in progress. The trajectory-related feature fields identify that the dataset contains trajectory processing results; their existence indicates that the processing flow has been completed and the data has been successfully merged. The trajectory index column stores the trajectory task index corresponding to each frame of data; its existence indicates that the columnar storage file has been updated with trajectory data.

[0096] Determining the processing status of the task execution segment based on the judgment result may include: When there is no temporary track directory, the metadata information file contains track-related feature fields, and the columnar storage file contains a track index column, it is determined to be in a completed state. When there is no temporary track directory, the metadata information file does not contain track-related feature fields, and the columnar storage file does not contain a track index column, it is determined to be a data loss state; When a temporary trajectory directory exists, the metadata information file does not contain trajectory-related feature fields, and the columnar storage file does not contain a trajectory index column, it is determined to be in the initial processing state. When a temporary trajectory directory exists, the metadata information file contains trajectory-related feature fields, and the columnar storage file contains trajectory index columns, it is determined to be in a rerun / overwrite state.

[0097] When the processing flow for each task execution segment is interrupted and then restarted, breakpoint resumption processing is performed based on the processing status of each task execution segment, which may include: When the processing flow for each task execution segment is interrupted and then restarted, for each task execution segment, if the processing status of the task execution segment is "completed", then the task execution segment is skipped; if the processing status of the task execution segment is "first processing", then processing continues from the interruption point or restarts; if the processing status of the task execution segment is "data lost", then an error is reported; if the processing status of the task execution segment is "rerun and overwrite", then the data corresponding to the task execution segment is deleted and reprocessed.

[0098] Therefore, by detecting whether a temporary trajectory directory with a preset name exists in the file system of the task execution segment, whether the feature field list of the metadata information file contains trajectory-related feature fields, and whether the columnar storage file contains a trajectory index column, and determining the processing status of the task execution segment based on the judgment results, and then when the processing flow of each task execution segment is interrupted and restarted, the method of resuming interrupted transmission based on the processing status of each task execution segment can infer the processing status of the task execution segment based on the signals corresponding to the temporary trajectory directory, the signals corresponding to the trajectory-related feature fields, and the signals corresponding to the trajectory index column, and further perform resuming transmission based on interrupted transmission, thereby improving fault tolerance and having higher reliability compared to resuming transmission based on a single state.

[0099] The trajectory processing pipeline processes each task segment independently, generating a line-based JSON file for each segment and storing it in a temporary trajectory directory. After processing, these scattered temporary files need to be merged into the standardized storage format of the source dataset to achieve unified management of trajectory data and raw data.

[0100] In the data reading phase, this invention employs a dual-path parsing strategy to address occasional line concatenation anomalies. Line concatenation anomalies refer to adjacent frames of data in a line-based JSON file being concatenated into a single data line due to missing newline characters. The dual path includes a fast path and a slow path.

[0101] Specifically, when reading each line of string in the line-based JSON format file corresponding to each task execution segment, the line of string is parsed using an integrity check parsing method. It is determined whether there is any remaining content after parsing. If so, a data frame is extracted one at a time from the beginning of the line of string using an incremental parsing method, and the data frame and its end position in the string are returned until the end of the line of string. A first list containing all extracted data frames is returned, and each data frame in the first list is processed frame by frame. If not, a list containing a single data frame is returned.

[0102] The fast path uses an integrity check parsing method to parse the string. The slow path uses an incremental parsing method to extract one data frame at a time from the beginning of the string and returns the data frame and its end position in the string until the end of the string. It then returns a first list containing all the extracted data frames and processes each data frame in the first list frame by frame.

[0103] In other words, the fast path processes data in normal format, while the slow path automatically repairs row concatenation anomalies, ensuring data integrity.

[0104] Example 2 Figure 2 This is a schematic flowchart illustrating a data generation method provided in an embodiment of the present invention. The method is applied to an electronic device. Specifically, the method includes the following steps: S210: Acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device.

[0105] S220: Based on the joint state data, calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device.

[0106] S230: The second transformation matrix from the world coordinate system to the base coordinate system is calculated based on motion data; S240: Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm is projected onto the video image to generate a two-dimensional trajectory.

[0107] The process of steps S210-S240 is the same as that of steps S110-S140 in Embodiment 1, and will not be repeated here.

[0108] S250: Samples are performed at the boundary frames of the atomic tasks to obtain multiple sample frames. The teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks have changed.

[0109] Because existing technologies lack methods for automatically generating VQA (Visual Question Answering) training data for trajectory understanding from robot teleoperation data, this invention constructs three types of labeled sample data for VQA, supporting three types of VQA tasks: trajectory coordinate prediction, trajectory overlay multiple-choice question, and atomic task planning multiple-choice question.

[0110] To construct the labeled sample data, multiple sample frames need to be obtained by sampling at the boundary frames of the atomic tasks in the teleoperation data. The teleoperation data includes N task execution segments, each of which includes multiple atomic tasks. The boundary frames are the frames where the atomic tasks change. For example, if atomic task 0 includes frames 0-199 and atomic task 1 includes frames 200-399, then the boundary frames are frames 0, 199, 200, and 399, and the sample frames are frames 0, 199, 200, and 399.

[0111] S260: Based on the two-dimensional trajectory of each sampling frame, generate corresponding labeled sample data for model training according to the type of visual question answering task.

[0112] After obtaining each sampling frame, the two-dimensional trajectory of each sampling frame can be obtained through steps S210-S240, and then the corresponding labeled sample data for model training can be generated according to the type of visual question answering task.

[0113] For three types of VQA tasks: Category 1: Trajectory Coordinate Prediction Tasks When the visual question answering task is a trajectory coordinate prediction task, step S260 may include: For each sampled frame, a first visual question answering task is constructed. The input of the first visual question answering task includes the video image of the two-dimensional trajectory of the sampled frame and the question text generated based on the preset question template, which contains the instructions of the first high-level task and the current atomic task corresponding to the sampled frame. Determine the future two-dimensional trajectory of the end effector corresponding to the sampled frame, and use it as the answer to the first visual question answering task for the sampled frame; Based on the input and answer of the first visual question-answering task in each sample frame, the first labeled sample data for model training is generated.

[0114] Since VQA training requires a large number of samples, and each sampled frame can represent an independent training sample, it is necessary to construct a first visual question answering task for each sampled frame. The input of the first visual question answering task includes the video image where the two-dimensional trajectory of the sampled frame is located and the question text generated based on the preset question template, which contains the instructions of the first high-level task and the current atomic task corresponding to the sampled frame.

[0115] High-level task instructions describe the overall goal, such as "pick up the cup and place it on the tray," while atomic tasks describe specific execution steps, such as "grab the cup." High-level task instructions provide what to do, and atomic tasks provide what is currently being done; combining the two allows the model to learn and understand the task hierarchy. High-level tasks and their contained atomic tasks are predefined.

[0116] Then, the future two-dimensional trajectory of the end effector corresponding to the sampling frame is determined. Since historical teleoperation data are used, the future two-dimensional trajectory of the end effector corresponding to the sampling frame can be directly used as the answer to the first visual question-and-answer task of the sampling frame.

[0117] Once the answer is obtained, the first labeled sample data for model training can be generated based on the input and answer of the first visual question-answering task in each sample frame.

[0118] Using the first labeled sample data as training data for the model, the model can be trained to learn the mapping from the video image where the two-dimensional trajectory of the current frame image, i.e. the sampled frame, is located, and the task description formed by the first high-level task instruction and the question text of the current atomic task to the future two-dimensional trajectory, so that the model can predict the two-dimensional trajectory coordinates according to the new scene during inference.

[0119] Therefore, for each sampled frame, a first visual question-answering task is constructed. The input of the first visual question-answering task includes the video image of the 2D trajectory of the sampled frame and the question text generated based on a preset question template, which includes the first high-level task instruction and the current atomic task corresponding to the sampled frame. This determines the future 2D trajectory of the end effector corresponding to the sampled frame and serves as the answer to the first visual question-answering task for that sampled frame. Based on the input and answer of the first visual question-answering task for each sampled frame, first labeled sample data for model training is generated. This achieves the goal of automatically generating first labeled sample data for model training corresponding to the trajectory coordinate prediction task based on the video image of the 2D trajectory generated from the robot's teleoperation data, eliminating the need for manual annotation and significantly reducing the cost of dataset construction. Simultaneously, through the trajectory coordinate prediction task, the model is trained to understand the reasonable operation trajectory in the current scene, thereby enabling it to predict the 2D trajectory according to new scenes.

[0120] Category 2: Trajectory Overlay Multiple Choice Questions When the visual question answering task is a trajectory overlay multiple-choice task, step S260 may include: For each sampled frame, the three-dimensional spatial position matrix of the end effector of the future frame is projected onto the video image of the sampled frame through the intrinsic parameter matrix of the image acquisition device of the sampled frame by cross-frame reprojection; The two-dimensional trajectory points obtained by projection are drawn onto the video image of the sampled frame to generate a trajectory overlay map; Construct a second visual question answering task, wherein the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampling frame and the second high-level task instruction; Based on the atomic task label corresponding to the teleoperation data in which the sampled frame is located, the first correct atomic task is determined and used as the first correct answer option; Based on the second high-level task instruction, the first correct atomic task, and the list of other atomic tasks under the second high-level task, generate multiple first interference options that are semantically similar but have different actions; The first correct answer option and multiple first interference options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the first correct answer option is used as the answer to the second visual question answering task of the sampled frame. Based on the input, multiple-choice options, and answers of the second visual question-answering task for each sample frame, second labeled sample data for model training is generated.

[0121] Specifically, by projecting the three-dimensional spatial position matrix of the end effector in a future frame onto the video image of the sampled frame through the intrinsic parameter matrix of the image acquisition device of the sampled frame via cross-frame reprojection, it can be: From the future frame range corresponding to the sampled frame, a preset number of reference frames are extracted using an equal-interval sampling method. For example, the preset number is 5, and the time window corresponding to the future frame range is 2 seconds.

[0122] For each reference frame, the three-dimensional spatial position matrix of the end effector in the reference frame, in the base coordinate system of the robotic arm, is projected onto the video image of the sampled frame using the intrinsic parameter matrix of the image acquisition device for that sampled frame. Then, the two-dimensional trajectory points obtained from the projection are plotted onto the video image of the sampled frame to generate a trajectory overlay map.

[0123] To construct training samples for the second type of VQA, a second visual question answering task needs to be constructed for each sampled frame. The input of the second visual question answering task includes the trajectory overlay map corresponding to the sampled frame and the second high-level task instruction.

[0124] Then, based on the atomic task label corresponding to the teleoperation data in which the sampled frame is located, the first correct atomic task is determined and used as the first correct answer option. The correct atomic task is the subtask stage in which the current sampled frame is located, which is a kind of semantic annotation information derived from the teleoperation data.

[0125] Based on the second high-level task instruction, the first correct atomic task, and the list of other atomic tasks under the second high-level task, multiple first interference options with similar semantics but different actions are generated. It is determined that the multiple choice question has reasonable difficulty. The first interference options are semantically similar to the first correct answer option corresponding to the first correct atomic task, but there are differences in specific operation actions, target objects, or execution order.

[0126] There are several ways to generate interference options, including but not limited to the following: The first type: Samples are taken from other stages of the second high-level task to obtain multiple first interference options that are semantically similar but have different actions.

[0127] The second type: Sample from subtasks of other high-level tasks at the same level as the second high-level task to obtain multiple first interference options that are semantically similar but have different actions.

[0128] The third type: Multiple first perturbation options with similar semantics but different actions are generated based on LLM (Large Language Model).

[0129] After obtaining the first interference option, the first correct answer option corresponding to the first correct atomic task is randomly arranged with multiple first interference options to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the first correct answer option is found as the answer to the second visual question answering task of the sampled frame.

[0130] Based on the input, multiple-choice options, and answers of the second visual question-answering task from each sample frame, second labeled sample data for model training is generated in a multimodal dialogue format.

[0131] Therefore, by projecting the 3D spatial position matrix of the future end effector onto the current video image through cross-frame reprojection, a trajectory overlay map is obtained as a visual cue. Semantically similar distracting options are generated to ensure the multiple-choice questions have reasonable difficulty, avoiding model learning through simple pattern matching. The second high-level task instructions are deeply integrated with each atomic task. Through atomic task multiple-choice questions, the model is trained to recognize the currently executed atomic task, enhancing its task state awareness. Finally, based on the input of the second visual question-and-answer task, the multiple-choice options, and the answers from each sampled frame, second labeled sample data for model training is generated. This achieves the goal of automatically generating second labeled sample data for model training corresponding to the trajectory overlay multiple-choice task based on the trajectory overlay map generated by cross-frame reprojection of robot teleoperation data. Simultaneously, through the trajectory overlay multiple-choice task, the model is trained to infer motion intent from trajectory morphology and select the correct atomic task from multiple options.

[0132] Category 3: Atomic Task Planning Multiple Choice Tasks When the visual question-answering task is an atomic task planning multiple-choice question task, step S260 may include: For each sampled frame, a third visual question answering task is constructed, wherein the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame and the third high-level task instruction; By analyzing motion behavior and semantic annotation, the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame is determined and used as the second correct answer option; Based on the third high-level task instruction, the second correct atomic task, and the list of other atomic tasks under the third high-level task, generate multiple second interference options that are semantically similar but have different actions. The second correct answer option and multiple second distractor options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the second correct answer option is used as the answer to the third visual question answering task of the sampled frame. Based on the input, multiple-choice options, and answers of the third visual question-answering task in each sample frame, third-labeled sample data for model training is generated.

[0133] The method for constructing the third labeled sample data is basically the same as that for constructing the second labeled sample data. The only difference is that the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampled frame, while the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame. The method for determining the correct atomic task is also different. It is through motion behavior analysis and semantic annotation that the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame is determined. The second correct atomic task is the task label. Therefore, task labels can be added based on the two-dimensional trajectory to construct a frame-level task label library. For other details, please refer to the construction of the second labeled sample data, which will not be repeated here.

[0134] Therefore, based on the third high-level task instruction, the second correct atomic task, and the list of other atomic tasks under the third high-level task, semantically similar interference options are generated to ensure that the multiple-choice questions have reasonable difficulty and avoid the model learning through simple pattern matching. The third high-level task instruction is deeply integrated with each atomic task. Through atomic task multiple-choice questions, the model is trained to identify the currently executing atomic task, enhancing its task state awareness. Finally, based on the input, multiple-choice options, and answers of the third visual question-answering task in each sample frame, third-labeled sample data for model training is generated. This achieves the goal of automatically generating third-labeled sample data for model training corresponding to the atomic task planning multiple-choice task based on the two-dimensional trajectory generated from the robot's teleoperation data. Simultaneously, through the atomic task planning multiple-choice task, the model is trained to infer the correct trajectory semantics from the two-dimensional trajectory and select the correct atomic task from multiple options.

[0135] In this embodiment of the invention, the three types of labeled sample data for VQA support three types of VQA tasks: trajectory coordinate prediction task, trajectory overlay multiple choice task, and atomic task planning multiple choice task.

[0136] in, Trajectory Coordinate Prediction Task: The input is a video image of the two-dimensional trajectory and a task description, and the answer is the future two-dimensional trajectory, which is used to predict the future two-dimensional trajectory based on a new scene; Trajectory Overlay Multiple Choice Question Task: The input is a trajectory overlay image, and the answer is the letter of the correct option in a multiple choice question. It is used to infer the motion intention from the trajectory shape and select the correct option corresponding to the motion intention from multiple options. The atomic task planning multiple-choice task, also known as the pure visual atomic task recognition task, involves inputting a two-dimensional trajectory and selecting the correct option letters from multiple-choice questions. The task is to infer the correct trajectory semantics from the two-dimensional trajectory and select the correct option corresponding to the trajectory semantics from multiple options.

[0137] Therefore, after generating the 2D trajectory, sampling is performed at the boundary frames of the atomic tasks to obtain multiple sample frames. The teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks change. Then, based on the 2D trajectory of each sample frame, corresponding labeled sample data for model training is generated according to the type of visual question answering task. This achieves the goal of transforming trajectory data into labeled sample data for model training corresponding to different types of visual question answering tasks, thereby improving the model's ability to understand robot operation trajectories.

[0138] Example 3 Figure 3 This is a schematic diagram of a robot end effector trajectory generation device provided in an embodiment of the present invention. (See attached diagram.) Figure 3 The robot end effector trajectory generation device provided in this embodiment of the invention may include: The first acquisition module 301 is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The first calculation module 302 is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data. The second calculation module 303 is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; The first trajectory projection module 304 is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device.

[0139] In one implementation, the motion data includes wheel speed data and steering angle data, and the second calculation module 303 includes: The body speed calculation submodule is used to obtain the body speed through inverse kinematics calculation based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data. The second transformation matrix calculation submodule is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system by numerical integration based on the body velocity.

[0140] In one implementation, the first trajectory projection module 304 includes: The merging submodule is used to merge the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device to obtain a normalized projection matrix; The multiplication submodule is used to multiply the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix to obtain homogeneous pixel coordinates; The normalized pixel coordinate determination submodule is used to perform perspective division on the homogeneous pixel coordinates to obtain normalized pixel coordinates; The drawing submodule is used to draw the two-dimensional trajectory points corresponding to the normalized pixel coordinates on the video image.

[0141] Therefore, the robot end effector trajectory generation device provided in this embodiment of the invention, based on a first transformation matrix, a second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, performs three coordinate transformations: the first transformation from the base coordinate system to the world coordinate system, the second transformation from the world coordinate system to the image acquisition device coordinate system, and the third transformation from the image acquisition device coordinate system to the image pixel plane. This maps the three-dimensional spatial position matrix of the end effector in the base coordinate system to the image pixel plane of the video image, generating a two-dimensional trajectory. By transforming the three-dimensional spatial position matrix of the end effector in the base coordinate system and the parameters of the image acquisition device to the world coordinate system through the first and second coordinate transformations, the two-dimensional trajectory obtained by the third coordinate transformation can correctly reflect the visual convergence effect when the robot moves, rather than rigidly moving with the robot as a whole, thus improving the accuracy of trajectory drawing. Furthermore, it can automatically generate two-dimensional trajectories based on teleoperation data, saving time and effort.

[0142] Example 4 Figure 4 This is a schematic diagram of a data generation device provided in an embodiment of the present invention. See also: Figure 4 The data generation apparatus provided in this embodiment of the invention may include: The second acquisition module 401 is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The third calculation module 402 is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the image acquisition device coordinate system based on the joint state data. The fourth calculation module 403 is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data; The second trajectory projection module 404 is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device. The sampling module 405 is used to perform sampling at the boundary frames of atomic tasks to obtain multiple sampling frames, wherein the teleoperation data includes multiple atomic tasks, and the boundary frames are frames where atomic tasks change. The labeled sample data generation module 406 is used to generate corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame, according to the type of visual question answering task.

[0143] Therefore, the data generation apparatus provided in this embodiment of the invention, after generating a two-dimensional trajectory, performs sampling at the boundary frames of the atomic tasks to obtain multiple sampling frames. The teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks change. Then, based on the two-dimensional trajectory of each sampling frame, corresponding labeled sample data for model training is generated according to the type of visual question-answering task. This achieves the purpose of converting trajectory data into labeled sample data for model training corresponding to different types of visual question-answering tasks, thereby improving the model's ability to understand the robot's operation trajectory.

[0144] In one implementation, when the visual question-answering task is a trajectory coordinate prediction task, the labeled sample data generation module 406 includes: The first construction submodule is used to construct a first visual question answering task for each sampled frame. The input of the first visual question answering task includes a video image of the two-dimensional trajectory of the sampled frame and a question text generated based on a preset question template, which includes instructions for a first high-level task and the current atomic task corresponding to the sampled frame. The first answer determination submodule is used to determine the future two-dimensional trajectory of the end effector corresponding to the sampling frame, and to serve as the answer to the first visual question answering task of the sampling frame; The first generation submodule is used to generate the first labeled sample data for model training based on the input and answer of the first visual question answering task in each sampled frame.

[0145] In one implementation, when the visual question-answering task is a trajectory overlay multiple-choice task, the labeled sample data generation module 406 includes: The projection submodule is used to project the three-dimensional spatial position matrix of the end effector of the future frame onto the video image of the sample frame through cross-frame reprojection for each sample frame. The trajectory overlay generation submodule is used to draw the two-dimensional trajectory points obtained by projection onto the video image of the sampled frame to generate a trajectory overlay. The second construction submodule is used to construct the second visual question answering task, wherein the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampling frame and the second high-level task instruction; The first correct answer option determination submodule is used to determine the first correct atomic task based on the atomic task label in the teleoperation data where the sampled frame is located, and use it as the first correct answer option; The first interference option generation submodule is used to generate multiple first interference options with similar semantics but different actions based on the second high-level task instruction, the first correct atomic task and the list of other atomic tasks under the second high-level task. The second answer determination submodule is used to randomly arrange the first correct answer option and multiple first interference options to form multiple choice options, and assign corresponding option identifiers, and find the option identifier corresponding to the first correct answer option as the answer to the second visual question answering task of the sampled frame; The second generation submodule is used to generate second labeled sample data for model training based on the input, selection options and answers of the second visual question answering task based on each sampled frame.

[0146] In one implementation, when the visual question-answering task is an atomic task planning multiple-choice question task, the labeled sample data generation module 406 includes: The third construction submodule is used to construct a third visual question answering task for each sampled frame, wherein the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame and the third high-level task instruction; The second correct answer option determination submodule is used to determine the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame through motion behavior analysis and semantic annotation, and to serve as the second correct answer option; The second interference option generation submodule is used to generate multiple second interference options with similar semantics but different actions based on the third high-level task instruction, the second correct atomic task and the list of other atomic tasks under the third high-level task. The third answer determination submodule is used to randomly arrange the second correct answer option and multiple second interference options to form multiple choice options, and assign corresponding option identifiers. The option identifier corresponding to the second correct answer option is found as the answer to the third visual question answering task of the sample frame. The third generation submodule is used to generate third labeled sample data for model training based on the input, selection options and answers of the third visual question answering task based on each sampled frame.

[0147] The above-described apparatus embodiments correspond to the method embodiments and have the same technical effects. For detailed explanations, please refer to the method embodiments. The apparatus embodiments are derived from the method embodiments; detailed explanations can be found in the method embodiments section, and will not be repeated here.

[0148] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0149] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating the trajectory of a robot end effector, characterized in that, include: Acquire teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device; Based on the joint state data, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device are calculated. A second transformation matrix from the world coordinate system to the base coordinate system is calculated based on the motion data, wherein the motion data is wheel speed data and steering angle data; Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system is projected onto the video image to generate a two-dimensional trajectory. The step of calculating the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data includes: Based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data, the body speed is obtained through inverse kinematics calculation. Based on the machine velocity, the second transformation matrix from the world coordinate system to the base coordinate system is calculated recursively through numerical integration.

2. The method as described in claim 1, characterized in that, The step of projecting the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video image based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device to generate a two-dimensional trajectory includes: The first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device are combined to obtain the normalized projection matrix; The homogeneous pixel coordinates are obtained by multiplying the three-dimensional spatial position matrix of the end effector in the world coordinate system with the normalized projection matrix. Perform perspective division on the homogeneous pixel coordinates to obtain normalized pixel coordinates; Draw the two-dimensional trajectory points corresponding to the normalized pixel coordinates on the video image.

3. A data generation method, characterized in that, include: Acquire teleoperation data of the robot, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device; Based on the joint state data, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device are calculated. The second transformation matrix from the world coordinate system to the base coordinate system is calculated based on the motion data, wherein the motion data includes wheel speed data and steering angle data. The step of calculating the second transformation matrix from the world coordinate system to the base coordinate system based on the motion data includes: Based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data, the body speed is obtained through inverse kinematics calculation. Based on the body velocity, the second transformation matrix from the world coordinate system to the base coordinate system is obtained by numerical integration recursively calculation. Based on the first transformation matrix, the second transformation matrix, and the intrinsic parameter matrix of the image acquisition device, the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm is projected onto the video image to generate a two-dimensional trajectory. Multiple sampling frames are obtained by sampling at the boundary frames of atomic tasks, wherein the teleoperation data includes multiple atomic tasks, and the boundary frames are the frames where the atomic tasks have changed; Based on the two-dimensional trajectory of each sampling frame, corresponding labeled sample data for model training are generated according to the type of visual question answering task.

4. The method as described in claim 3, characterized in that, When the visual question answering task is a trajectory coordinate prediction task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame according to the type of visual question answering task includes: For each sampled frame, a first visual question answering task is constructed, wherein the input of the first visual question answering task includes the video image of the two-dimensional trajectory of the sampled frame and the question text generated based on the preset question template, which includes the instructions of the first high-level task and the current atomic task corresponding to the sampled frame; Determine the future two-dimensional trajectory of the end effector corresponding to the sampling frame, and use it as the answer to the first visual question answering task of the sampling frame; Based on the input and answer of the first visual question-answering task in each sample frame, the first labeled sample data for model training is generated.

5. The method as described in claim 3, characterized in that, When the visual question answering task is a trajectory overlay multiple-choice task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectories of each sampling frame according to the type of visual question answering task includes: For each sampled frame, the three-dimensional spatial position matrix of the end effector of the future frame is projected onto the video image of the sampled frame through the intrinsic parameter matrix of the image acquisition device of the sampled frame by cross-frame reprojection; The two-dimensional trajectory points obtained by projection are drawn onto the video image of the sampled frame to generate a trajectory overlay map; Construct a second visual question answering task, wherein the input of the second visual question answering task includes the trajectory overlay map corresponding to the sampling frame and the second high-level task instruction; Based on the atomic task label corresponding to the teleoperation data in which the sampled frame is located, the first correct atomic task is determined and used as the first correct answer option; Based on the second high-level task instruction, the first correct atomic task, and the list of other atomic tasks under the second high-level task, generate multiple first interference options that are semantically similar but have different actions; The first correct answer option and multiple first interference options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the first correct answer option is used as the answer to the second visual question answering task of the sampled frame. Based on the input, multiple-choice options, and answers of the second visual question-answering task for each sample frame, second labeled sample data for model training is generated.

6. The method as described in claim 3, characterized in that, When the visual question answering task is an atomic task planning multiple-choice question task, the step of generating corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame according to the type of visual question answering task includes: For each sampled frame, a third visual question answering task is constructed, wherein the input of the third visual question answering task includes the two-dimensional trajectory of the sampled frame and the third high-level task instruction; By analyzing motion behavior and semantic annotation, the second correct atomic task corresponding to the two-dimensional trajectory of the sampled frame is determined and used as the second correct answer option; Based on the third high-level task instruction, the second correct atomic task, and the list of other atomic tasks under the third high-level task, generate multiple second interference options that are semantically similar but have different actions; The second correct answer option and multiple second distractor options are randomly arranged to form multiple choice options, and corresponding option identifiers are assigned. The option identifier corresponding to the second correct answer option is used as the answer to the third visual question answering task of the sampled frame. Based on the input, multiple-choice options, and answers of the third visual question-answering task in each sample frame, third-labeled sample data for model training is generated.

7. A robot end effector trajectory generation device, characterized in that, include: The first acquisition module is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The first calculation module is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data. The second calculation module is used to calculate a second transformation matrix from the world coordinate system to the base coordinate system based on the motion data, wherein the motion data is wheel speed data and steering angle data; The first trajectory projection module is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device. The second computing module includes: The body speed calculation submodule is used to obtain the body speed through inverse kinematics calculation based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data. The second transformation matrix calculation submodule is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system by numerical integration based on the body velocity.

8. A data generation apparatus, characterized in that, include: The second acquisition module is used to acquire the robot's teleoperation data, wherein the teleoperation data includes at least joint state data, motion data of the mobile chassis, and video images acquired by the image acquisition device. The third calculation module is used to calculate the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm and the first transformation matrix from the base coordinate system to the coordinate system of the image acquisition device based on the joint state data. The fourth calculation module is used to calculate a second transformation matrix from the world coordinate system to the base coordinate system based on the motion data, wherein the motion data are wheel speed data and steering angle data. The fourth calculation module includes: The body speed calculation submodule is used to obtain the body speed through inverse kinematics calculation based on the kinematic model of the mobile chassis, the wheel speed data, and the steering angle data. The second transformation matrix calculation submodule is used to calculate the second transformation matrix from the world coordinate system to the base coordinate system based on the body velocity through numerical integration recursively. The second trajectory projection module is used to project the three-dimensional spatial position matrix of the end effector in the base coordinate system of the robotic arm onto the video image to generate a two-dimensional trajectory based on the first transformation matrix, the second transformation matrix and the intrinsic parameter matrix of the image acquisition device. The sampling module is used to perform sampling at the boundary frames of atomic tasks to obtain multiple sampling frames, wherein the teleoperation data includes multiple atomic tasks, and the boundary frames are frames where atomic tasks have changed; The labeled sample data generation module is used to generate corresponding labeled sample data for model training based on the two-dimensional trajectory of each sampling frame, according to the type of visual question answering task.

Images