A method, device and storage medium for mechanical arm data processing

Abstract

This application relates to a method, device, and storage medium for robotic arm data processing. It involves acquiring the current pose and configuration information of the robotic arm, and determining its dynamic reachable space based on this information. The dynamic reachable space is then mapped onto a user interface for visualization, including a progress bar and a speedometer to show the progress of the reachable space. If the reachable progress is within a specified range, the robotic arm continues data acquisition within that range. If the reachable progress is outside the specified range, the acquired data outside the range is cleaned. This standardizes the actions of the data acquisition operator, ensures that the acquired data can be subsequently executed by the robotic arm, avoids the generation of invalid data, quantifies data quality, and filters valid data to ensure that the filtered data meets the robotic arm's execution requirements.

Description

Technical Field

[0001] This application relates primarily to the field of robotics technology, and in particular to a method, apparatus, and storage medium for processing data from a robotic arm. Background Technology

[0002] The upper limbs of embodied intelligent robots typically consist of a robotic arm and an end effector (gripper or dexterous hand). The robotic arm is rotatable thanks to multiple high-precision motors, while the end effector is mounted at the end of the robotic arm. To monitor task completion, cameras are placed at both the main viewpoint (global observation) and the wrist (local task observation). Traditional upper limb task data acquisition usually involves a data collector remotely controlling the robotic arm and end effector to complete the task via command input devices such as master-slave arms or VR. A host computer simultaneously records video from the main viewpoint and wrist viewpoint, as well as the time series of the robotic arm joint angles, forming a task data set. However, this approach is highly dependent on the target robotic arm hardware, and the data from different robotic arms is device-independent, making it impossible to share and reuse data.

[0003] To address the aforementioned issues, the Unified Operation Interaction (UMI) data acquisition system employs a method of recording the end effector pose (EEF) of the robotic arm instead of joint angles, thus eliminating reliance on physical robotic arm hardware. The UMI system consists of a handheld gripper, a camera, and a positioner. The data acquisition operator directly holds the gripper to complete the task, while the positioner collects real-time EEF data and video of the hand, which is then recorded by a host computer. Subsequently, after the Visual Language Action Model (VLA) or Imitation Learning algorithm predicts the EEF, it needs to be converted into joint angles using an inverse kinematics (IK) algorithm before the robotic arm can execute the task.

[0004] However, existing UMI data acquisition systems do not take into account the execution problems caused by different robotic arm configurations when converting EEF into joint rotation angles. This results in problems such as joint overspeed, unnatural motion jamming, and crossing singularities in the robotic arm configuration after trajectory conversion, which affect the performance of the robotic arm task. Summary of the Invention

[0005] One objective of this application is to provide a method, apparatus, and storage medium for processing data from a robotic arm, in order to solve the problem that existing UMI data acquisition methods do not consider the feasibility of robotic arm execution, resulting in invalid data.

[0006] According to one aspect of this application, a method for processing data of a robotic arm is provided. The method includes: acquiring current pose information and configuration information of the robotic arm; determining the dynamic reachable space of the robotic arm based on the current pose information and configuration information; mapping the dynamic reachable space of the robotic arm onto a user interface for display in a visual graphic, the visual graphic including a progress bar and a speedometer for displaying the reachability progress of the dynamic reachable space; if the reachability progress is within a target area, driving the robotic arm to continue performing data acquisition operations within the target area; if the reachability progress is not within the target area, cleaning the acquired data that is not within the target area.

[0007] Optionally, determining the dynamic reachability space of the robotic arm based on the current pose information and configuration information includes: determining the joint state parameters of the robotic arm based on the current pose information and configuration information; determining a Jacobian matrix based on the joint state parameters to establish the correlation between the joint velocities and the end effector velocities of the robotic arm; setting constraints on the joint velocities; determining the parameter set of the end effector motion based on the Jacobian matrix and the constraints; and converting the parameter set of the end effector motion into a corresponding geometric space based on the Jacobian matrix to determine the dynamic reachability space of the robotic arm.

[0008] Optionally, mapping the dynamic reachability space of the robotic arm onto the user interface for visual display includes: determining multiple preset directions in the dynamic reachability space and the maximum reachable motion index in each preset direction; normalizing the maximum reachable motion index to obtain a standard reachable capacity index in each preset direction; and mapping the standard reachable capacity index onto the user interface for visual display.

[0009] Optionally, the step of cleaning the collected data that is not within the target area includes: extracting the target pose information of each frame of data in the collected data, determining joint angle parameters based on the target pose information; determining the pose error according to the joint angle parameters, filtering the collected data according to the pose error and a preset error threshold to obtain valid data, and cleaning the valid data.

[0010] Optionally, the step of cleaning the valid data includes: determining the end effector speed of the robotic arm based on the joint angle parameters; determining a speed code table based on the end effector speed of the robotic arm, wherein the speed code table is used to display the speed percentage by color partition; and cleaning the valid data based on the color currently displayed on the speed code table.

[0011] Optionally, the velocity code table includes a linear velocity code table and an angular velocity code table. Determining the velocity code table based on the joint angle parameters includes: determining the end effector velocity of the robotic arm based on the joint angle parameters; determining linear velocity components and angular velocity components based on the components of the end effector velocity of the robotic arm in different spatial dimensions; generating a linear velocity ratio based on the linear velocity components and a preset linear velocity threshold; and mapping the linear velocity ratio to a linear velocity code table; generating an angular velocity ratio based on the angular velocity components and a preset angular velocity threshold; and mapping the angular velocity ratio to an angular velocity code table.

[0012] Optionally, the step of cleaning the valid data according to the color currently displayed on the speed code table includes: when the color currently displayed on the speed code table is the target color, correcting the end effector speed of the robotic arm by time scaling to complete the cleaning of the valid data.

[0013] Optionally, the method further includes: acquiring the current frame data of the collected robotic arm data, and acquiring the dynamic reachable space corresponding to the current frame data; determining whether there is frame data located within the dynamic reachable space in a preset number of subsequent frame data; if not, discarding the collected data; if so, marking the frame closest to the current frame that meets the conditions as the target frame, discarding all frame data from the current frame to the frame preceding the target frame, and using the target frame as the new current frame, until the processing of all frame data in the collected data is completed.

[0014] According to another aspect of this application, an electronic device is also provided, comprising: one or more processors; and a memory storing computer-readable instructions that, when executed, cause the processor to perform operations as described above.

[0015] According to another aspect of this application, a computer-readable storage medium is also provided, having stored thereon computer-readable instructions that can be executed by a processor to implement the method described above.

[0016] Compared with existing technologies, this application obtains the current pose and configuration information of the robotic arm, and determines the dynamic reachable space of the robotic arm based on the current pose and configuration information; the dynamic reachable space of the robotic arm is mapped onto the user interface and displayed as a visual graphic, including a progress bar and a speedometer, to show the reachability progress of the dynamic reachable space; if the reachability progress is within the target area, the robotic arm is driven to continue performing data collection operations within the target area; if the reachability progress is not within the target area, the collected data outside the target area is cleaned. This standardizes the actions of the data collector, ensures that the collected data can be subsequently executed by the robotic arm, avoids the generation of invalid data, and quantifies data quality, filters valid data, and ensures that the filtered data meets the execution requirements of the robotic arm. Attached Figure Description

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings, wherein:

[0018] Figure 1 A flowchart of a robotic arm data processing method according to one aspect of this application is shown;

[0019] Figure 2 A schematic diagram of a user interface showing a virtual robotic arm and visual graphics is shown in one embodiment of this application;

[0020] Figure 3 This document illustrates a flowchart of the cleaning process for collected UMI data in one embodiment of this application.

[0021] Figure 4 A schematic diagram of a frame of an electronic device provided according to another aspect of this application is shown.

[0022] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein, and therefore this application is not limited to the specific embodiments disclosed below.

[0025] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0026] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0027] (1) Universal Manipulation Interface (UMI) is an intermediate layer interface or protocol designed to standardize robot operation instructions, allowing upper-layer algorithms (such as AI models) to send uniform, abstract action instructions, such as "move forward 10 centimeters", without having to pay attention to the specific robot model at the bottom.

[0028] (2) End Effect (EEF) refers to the position and orientation of the end effector (such as gripper, suction cup or tool) of a robot arm in three-dimensional space. EEF is usually described by 6 degrees of freedom (6-DoF), namely position, which is the coordinate on the X, Y and Z axes, and orientation, which is usually represented by Euler angles or quaternions, describing the direction in which the end effector is facing.

[0029] (3) Inverse Kinematics (IK) is a core computational process in robot kinematics. Given the target pose of the end effector (EEF), for example, the position of the hand reaching the apple on the table, the angles that each joint (such as the shoulder, elbow, and wrist) needs to rotate are deduced through mathematical calculation.

[0030] (4) Vision Language Action Model (VLA) is a large artificial intelligence model that combines vision, language and action capabilities. It can understand human natural language commands and output specific action sequences of the robot (such as movement path and grasping command) based on the images seen by the camera.

[0031] The development of large-scale embodied intelligent manipulation models is rapid, and their training heavily relies on massive amounts of embodied robotic arm manipulation data. Existing Unified Manipulation Interaction (UMI) data acquisition methods use handheld devices to simulate end effectors, completing data acquisition without relying on a physical robotic arm, effectively saving hardware costs. Furthermore, because they do not require waiting for physical execution by the robotic arm, their acquisition speed is significantly faster than traditional methods such as master-slave arms. However, existing UMI methods lack consideration for the physical configuration and kinematic constraints of the robotic arm during the acquisition process, resulting in the acquired motion trajectories often exceeding the robotic arm's capabilities and generating a large amount of invalid data that cannot be actually executed by the robotic arm.

[0032] To address the aforementioned issues, this application proposes a method for processing data from a robotic arm, which enables visualized operational constraints and standardizes the operational behavior of data collectors, thereby ensuring the feasibility and effectiveness of the collected data during subsequent robotic arm execution.

[0033] Figure 1 The diagram shows a flowchart of a robotic arm data processing method according to one aspect of this application, the method including steps S11 to S14.

[0034] Step S11: Obtain the current pose information and configuration information of the robotic arm, and determine the dynamic reachable space of the robotic arm based on the current pose information and configuration information.

[0035] Based on the current motion state (current pose information) and hardware parameters (configuration information) of the robotic arm, the dynamic reachable space of the robotic arm in the current state is determined through kinematic modeling and constraint calculation. The pose information of the robotic arm includes the real-time position and attitude data of the joints and end effector, while the configuration information includes hardware parameters that determine the robotic arm's motion capabilities, such as the number of joints, joint range of motion, link length, and joint type. Based on the collected pose and configuration information of the robotic arm, and combined with kinematic theory, the set of spatial positions that the robotic arm can reach in the current pose is determined, i.e., the dynamic reachable space, such as the reachability ellipsoid.

[0036] Step S12: Map the dynamic reachable space of the robotic arm onto the user interface and display it as a visual graphic. The visual graphic includes a progress bar and a speedometer to display the reachability progress of the dynamic reachable space.

[0037] Dynamic reachability space is typically an abstract spatial parameter. Transforming the abstract dynamic reachability space of a robotic arm into intuitive visual information allows operators to easily adjust the operating range and actions by observing this visual information during operation. The calculated dynamic reachability space can be mapped to the user interface, presented as a 3D model, boundary contour, color-coded partitions, and other visual graphics, providing real-time feedback on the reachability progress. For example, colors can distinguish between reachable and inaccessible areas, or dynamic markers can display the distance between the current operating position and the reachable boundary. A speedometer can display the permissible movement speed of the robotic arm within the reachable area in stages, using numerical values, color blocks, and scale markings to differentiate safe speed ranges for different spatial positions.

[0038] Step S13: If the achievable progress is within the target area, then drive the robotic arm to continue performing data collection operations within the target area.

[0039] Whether the achievable progress falls within the target range is used as the criterion for judging the validity of the collected data. The target range can be a preset safety threshold interval, which can be adjusted according to the configuration of the robotic arm and the task requirements. If the operation within the target range can be stably executed by the robotic arm subsequently, it means that the collected data is valid.

[0040] When the achievable progress corresponding to the current operation is detected to be within the preset target area, it indicates that the current virtual robotic arm posture and operation direction meet the robotic arm execution requirements. At this time, no intervention is required. The virtual robotic arm is directly driven to continue to replicate the data acquisition operator's operation within the target area, and simultaneously completes the collection and recording of information such as EEF data and image data.

[0041] Step S14: If the achievable progress is not within the target area, then the collected data that is not within the target area is cleaned.

[0042] When the achievable progress is detected to be outside the target area, it indicates that the current operation has approached or entered the non-executable area of ​​the robotic arm, such as near a singularity, exceeding the joint speed limit, or approaching the workspace boundary. The corresponding data collection action cannot be subsequently executed by the robotic arm. At this time, a data cleaning mechanism will be triggered to filter the data collected under the abnormal scenario of "not within the target area," selecting cleanable and valid data. If cleaning is not possible, invalid data will be discarded.

[0043] In one embodiment of this application, in step S11, the joint state parameters of the robotic arm are determined based on the current pose information and configuration information; a Jacobian matrix is ​​determined based on the joint state parameters to establish the correlation between the joint velocity and the end effector velocity of the robotic arm; constraints on the joint velocity are set, and a set of parameters for the end effector motion is determined based on the Jacobian matrix and the constraints; the set of parameters for the end effector motion is transformed into a corresponding geometric space based on the Jacobian matrix to determine the dynamic reachability space of the robotic arm.

[0044] In any motion state of the robotic arm, the motion capability of the end effector (such as a gripper) under the current configuration can be analyzed using the Jacobian matrix, with a dynamic reachability space (such as an reachability ellipsoid) as the geometric representation. Specifically, by using the current pose information of the robotic arm (such as the real-time position and attitude of the end effector or joints) and configuration information (such as hardware parameters such as the number of joints, range of motion, and link dimensions), the joint state parameters of the robotic arm can be extracted. This allows the calculation of the Jacobian matrix of the end effector, thereby establishing a mapping relationship between joint velocities and end effector velocities.

[0045] The joint movements of a robotic arm have physical limits, such as maximum rotational speed and acceleration threshold. By setting constraints on joint velocities, safe execution states are selected. Then, using the Jacobian matrix, the set of end-effector motion parameters corresponding to the executable joint movements is further derived, such as the set of all achievable end-effector velocities and directions. Finally, the set of end-effector motion parameters is transformed into a specific range in three-dimensional geometric space, i.e., the dynamic reachable space of the robotic arm in its current state.

[0046] In practical applications, given the current joint angle vector of the robotic arm Where n is the number of degrees of freedom of the robotic arm's joints. Let represent the rotation angle of the i-th joint, in rad. Based on the forward kinematics model of the robotic arm, calculate the Jacobian matrix of the end effector in the base coordinate system. It describes the mapping relationship between joint velocity and end-effector velocity (linear velocity and angular velocity), and the formula is as follows:

[0047]

[0048] in , represents the joint angular velocity vector. This represents the angular velocity of the i-th joint, in rad / s; The generalized velocity vector of the end effector is 6-dimensional, where This represents the linear velocity components of the terminal in the X, Y, and Z axes of the base coordinate system. This represents the angular velocity components of the end effector around the X, Y, and Z axes of the base coordinate system. Using the above formula, the known joint angular velocities can be obtained. The end-effector generalized velocity can be calculated directly, or the joint velocity can be deduced from the end-effector velocity through inverse mapping.

[0049] Next, physical constraints are imposed on the joint velocities based on the speed upper limit of the robotic arm's hardware specifications. For example, if the joint velocity norm is limited, two specification forms can be selected. Form one is a single-joint velocity upper limit constraint:

[0050]

[0051] in The maximum allowable angular velocity of the i-th joint is given by the robot arm specifications. Form two is the joint velocity normalization constraint:

[0052]

[0053] The joint velocities are normalized to evaluate robotic arms with different joint specifications. Under the above constraints, the set of all achievable generalized velocities of the end effector is as follows:

[0054]

[0055] in Let represent the reachable set of end-effector velocities, that is, the set of all generalized velocities that the end-effector can achieve while satisfying joint velocity constraints. The boundary of this set in the 6-dimensional end-effector velocity space is an ellipsoid, and the ellipsoidal inequality is:

[0056]

[0057] when When reversible, the robotic arm is in a non-singular configuration, and the assembly... For a standard ellipsoid; when When irreversible, the robotic arm approaches a singular configuration, the matrix rank degenerates, and the ellipsoid degenerates accordingly, meaning that the robotic arm loses its motion capabilities in some dimensions.

[0058] right Eigenvalue decomposition yields the principal axis direction and semi-axis length of the ellipsoid. The principal axis direction represents the main direction of motion capability; the semi-axis length directly reflects the maximum achievable velocity of the end effector along that principal axis direction. A longer semi-axis indicates a greater achievable velocity and stronger motion capability in that direction. When the robotic arm approaches a singularity configuration, the Jacobian matrix... The rank decreases and some eigenvalues ​​tend to 0, corresponding to a significant shortening or even a tendency to 0 of the semi-axis length, indicating that the end effector's degree of freedom of motion in that direction is restricted, resulting in loss or extremely weak motion capability.

[0059] In one embodiment of this application, in step S12, multiple preset directions in the dynamic reachability space and the maximum reachability motion index in each preset direction are determined; the maximum reachability motion index is normalized to obtain a standard reachability capacity index in each preset direction, and the standard reachability capacity index is mapped onto the user interface for visual graphical display.

[0060] First, multiple preset directions corresponding to the dynamic reachability space are determined, such as six preset operation directions (X / Y / Z spatial translation directions and Roll / Pitch / Yaw rotational directions around the axis). Each preset operation direction corresponds to a motion capability component in one dimension of the dynamic reachability space. Next, based on the geometric features of the dynamic reachability space derived from joint state parameters, the maximum reachable motion index corresponding to the motion capability component in each preset operation direction is extracted. This index directly reflects the executable margin of the robotic arm in the corresponding direction. By calculating the ratio of the maximum reachable motion index to a preset reference speed and performing amplitude limiting processing for normalization, a standard reachability index ranging from 0 to 1 is finally obtained.

[0061] Finally, the standard reachable capacity index corresponding to each preset operation direction is mapped to a visual display element in the user interface, such as a progress bar, bar, ring bar, or a 3D marker in the simulation platform. By displaying the dynamic changes of the length, height, arc length, or transparency of the element with the standard reachable capacity index, the executable margin in each direction and the approaching state of the robot arm's workspace boundary are presented intuitively, thereby assisting data acquisition personnel in standardizing their operation actions.

[0062] In practical applications, to help data acquisition personnel intuitively perceive the executable margin under the current configuration of the robotic arm, instead of drawing a complete 6-dimensional reachability ellipsoid on the screen, the "allowed velocity projection" of the ellipsoid on the 6 commonly used operation directions (X / Y / Z translation direction and Roll / Pitch / Yaw rotation direction) is mapped as 6 progress bars (or bar bars, ring bars) to achieve an intuitive presentation of the executable margin.

[0063] Unit vector for each common operation direction (Corresponding to X / Y / Z / Roll / Pitch / Yaw respectively), calculate the maximum achievable velocity scalar in this direction, using the following formula:

[0064]

[0065] in Constraints are set for joint velocities to ensure that the calculation results conform to the motion limits of the robotic arm hardware. The calculated maximum achievable velocity scalar is then used. The normalization process is performed using the following formula:

[0066]

[0067] in Indicates the standard's operability index. The `clip` function represents a preset reference speed and limits the result to a range of 0 to 1. It directly maps the normalized standard operability index to the visual attributes of a progress bar (or bar chart, circular bar), including length, height, arc length, and transparency. Higher index values ​​correspond to a longer length / height / arc length and lower transparency for the corresponding visual element, intuitively reflecting stronger movement capability in that direction; conversely, lower values ​​indicate weaker movement capability.

[0068] exist Figure 2 The illustrated embodiment demonstrates a user interface for a simulation platform, including a virtual robotic arm and visual graphics. During data acquisition, the software generates a virtual robotic arm (e.g., ...). Figure 2 The virtual arm in the middle executes the data acquisition operator's actions in real time and provides visual elements to standardize the operator's input instructions. In actual engineering, it can directly output a 6-dimensional "operability index" array containing XYZ translation and RPY rotation dimensions, and visualize it in a simulation platform (such as RViz) through marker cylinders (corresponding to the translation direction) and rings (corresponding to the rotation direction), thereby transforming the "executability margin of the ellipsoid in a specified direction" into a concrete visual signal.

[0069] Specifically, Figure 2 In the interface, A1, A2, A3, and R1, R2, R3 are all visual progress bars. A1 corresponds to the X-axis translation, A2 to the Y-axis translation, and A3 to the Z-axis translation, appearing as cylindrical shapes in the simulation platform. R1 corresponds to the Roll direction, R2 to the Pitch direction, and R3 to the Yaw direction, appearing as circular shapes in the simulation platform. A1, A2, A3, and R1, R2, R3 represent the executable margins of the virtual robotic arm in the middle of the interface in the six dimensions of XYZ translation and RPY rotation. Different colors can be used to mark the progress bars; for example, yellow for A1, indigo for A2, and magenta for A3. When the data acquisition operator's actions cause the virtual robotic arm to gradually approach the executable boundary, such as approaching a singularity, approaching the joint speed limit, approaching the workspace boundary, or when a principal axis of the operability ellipsoid is significantly shortened, the progress bar of the corresponding color will gradually decrease from "full / long arc" to "short arc / space". When it enters the non-executable space, the progress bar drops to 0 or becomes a warning state (e.g., flashing / highlighting / red border), prompting the data acquisition operator to immediately retract or change the direction of action.

[0070] Each visual progress bar can simultaneously assist data acquisition personnel in judging the distance to the workspace boundary. When the gripper approaches the workspace boundary, the progress bar in the corresponding direction will drop significantly, indicating that the data acquisition personnel are about to enter an unexecutable area and need to promptly standardize their operations. If the progress bar in a certain dimension shortens significantly, the data acquisition personnel should reduce the motion component along that direction. For example, reduce advancement along that axis, switch to lateral movement, or change the posture before advancement, or adjust the robot arm's posture to an area with a longer progress bar (i.e., a larger ellipsoid with greater mobility) before continuing the task. Especially when the robot arm is in a singular configuration, its mobility will be lost, causing the reachability ellipsoid to shrink and the progress bars in all directions to shorten generally. In this case, the data acquisition personnel need to immediately adjust the posture and return to an area with a larger ellipsoid before performing the action to avoid irreversible solutions, joint overspeed, or configurational abrupt changes that could lead to invalid data.

[0071] In one embodiment of this application, in step S14, target pose information of each frame of data in the acquired data is extracted, joint angle parameters are determined based on the target pose information, pose error is determined according to the joint angle parameters, the acquired data is filtered according to the pose error and a preset error threshold to obtain valid data, and the valid data is cleaned.

[0072] For the collected data that does not meet the standards, the target pose information of each frame is first extracted and converted into joint angle parameters of the robotic arm. The pose error of the robotic arm is calculated based on the joint angle parameters and compared with a preset error threshold. If the pose error of a frame exceeds the preset threshold, it indicates that the end effector pose corresponding to that frame cannot be achieved or stably executed on the target robotic arm, and is therefore deemed invalid data. If the error does not exceed the threshold, it is deemed valid data. Only the filtered valid data is retained; invalid single-frame data or consecutive invalid trajectory segments are removed, and the valid data is cleaned, thus completing the data cleaning process.

[0073] In practical applications, the input data consists of the timestamp sequence t corresponding to each robotic arm motion data point, and the end-target pose of each frame of data. The target pose includes position and attitude information, derived from the EEF trajectory recorded by the UMI system. For the target pose at the end of each frame... The inverse kinematics (IK) solver of the target robotic arm is invoked to calculate the corresponding joint angle solutions. .

[0074] To ensure the stability and trajectory continuity of joint movements, a continuous solution strategy is adopted in the actual solution process: joint angle solutions from the previous frame of data. As the initial value (seed value) for solving the current frame, if the same end pose... If multiple joint angle solutions exist, the solution from the previous frame is selected. The solution with the minimum distance, for example, by calculating the least 2 norm. This needs to be determined to avoid sudden changes in joint angles.

[0075] Find the joint angle solution Afterwards, Substituting the forward kinematics equations of the robotic arm, the actual reachable pose of the end effector is calculated. Next, the original target pose is calculated. With actual reachable pose The deviation between them, that is, the position error is calculated separately. With attitude error Position error is calculated from the original position vector. With the predicted location vector Distance between Attitude error The rotation difference of the rotation matrix can be used. It can be calculated using axis-angle representation or quaternion distance.

[0076] Finally, the calculated error value is compared with the preset error threshold. If the position error or attitude error exceeds the corresponding threshold, the pose corresponding to the frame of data is determined to be unreachable or unstable on the target robotic arm, and the frame or trajectory segment is discarded as invalid data; if both the position error and attitude error are less than the corresponding threshold, the frame of data is determined to be reachable and valid data, and proceeds to the subsequent data cleaning process.

[0077] Furthermore, the end effector speed of the robotic arm is determined based on the joint angle parameters, and the speed code table is determined based on the end effector speed of the robotic arm, wherein the speed code table is used to display the speed percentage by color partition; the valid data is cleaned according to the color currently displayed on the speed code table.

[0078] For valid data that has been determined to be reachable after pose error verification, further speed evaluation can be performed to determine the data to be cleaned. Based on the joint angle parameters of the robotic arm, the speed of the robotic arm's end effector is obtained through kinematic calculations. The speed is visualized using a speedometer, with the upper limit of joint speed set in the robotic arm hardware specifications as a threshold. The ratio of the actual speed to the upper limit is divided into different intervals and marked with different colors. For example, green can represent the speed within the threshold range (compliant), yellow can represent the speed approaching the speed threshold (warning required), and red can represent the speed exceeding the speed threshold (overspeeding). Color-coded partitions quickly identify the current end effector speed status of the robotic arm.

[0079] Based on the color of the speedometer, valid data undergoes a secondary filtering process. For example, yellow (close to the threshold) data and green (compliant) data do not require cleaning, while red (speeding) data are valid data that are "speeding only but achievable" and can be further cleaned.

[0080] In practical applications, the joint angle data of consecutive frames is first extracted. (Current frame) and (Previous frame), based on the time interval between two frames of data The joint angular velocity is estimated using the first-order difference method, and the calculation formula is as follows:

[0081]

[0082] Since sensor noise can cause spikes in angular velocity values, affecting the accuracy of the judgment, the estimated values ​​can be modified accordingly. Further processing, such as moving average or low-pass filtering, is performed to smooth the angular velocity curve and remove abnormal spike signals.

[0083] angular velocity of each joint (i is the joint number) and the maximum speed limit of a single joint given in the robotic arm specifications. Perform a comparison one by one; if any joint satisfies... If a frame is found to be speeding, it is determined that the data exceeds the speed limit. The speed status is associated with the speed code table. Frames that are determined to be speeding are displayed in red on the speed code table, indicating that the data frame has passed the pose error check and only has a speeding problem, and subsequent data cleaning is performed.

[0084] In one embodiment of this application, the velocity code table includes a linear velocity code table and an angular velocity code table. The end effector velocity of the robotic arm is determined based on the joint angle parameters. Linear velocity components and angular velocity components are determined based on the components of the end effector velocity of the robotic arm in different spatial dimensions. A linear velocity ratio is generated based on the linear velocity components and a preset linear velocity threshold, and the linear velocity ratio is mapped to a linear velocity code table. An angular velocity ratio is generated based on the angular velocity components and a preset angular velocity threshold, and the angular velocity ratio is mapped to an angular velocity code table.

[0085] The velocity code table can include a linear velocity code table and an angular velocity code table. The linear velocity code table represents the translational velocity of the robotic arm's end effector in space, and the angular velocity code table represents the rotational velocity of the end effector around its axis. Specifically, based on joint angle parameters, the comprehensive velocity of the robotic arm's end effector is obtained through kinematic calculations. Then, based on a Cartesian coordinate system, the comprehensive velocity of the end effector is decomposed into components in different spatial dimensions, thereby extracting the linear velocity component (the superposition of velocities in the X, Y, and Z translational directions) and the angular velocity component (the superposition of velocities in the Roll, Pitch, and Yaw rotational directions).

[0086] The linear velocity component is compared with a preset linear velocity threshold to obtain the linear velocity ratio (i.e., the percentage of the actual linear velocity relative to the threshold). This ratio is then mapped to the color and scale of the linear velocity codebook. Similarly, the angular velocity component is compared with a preset angular velocity threshold to obtain the angular velocity ratio, which is then mapped to the angular velocity codebook. Through ratio quantization and color visualization, the velocity status becomes more intuitive.

[0087] In practical applications, the velocity of the endpoint in Cartesian space can be selected as the visualization object, specifically including the magnitude of the linear velocity. With the magnitude of angular velocity This directly corresponds to the intuitive operation of the data acquisition operator holding the gripper, making it easy for the data acquisition operator to adjust the operation in real time according to the code table prompts.

[0088] Meanwhile, to ensure the executability and safety of the robotic arm, a dual constraint mechanism of "front-end visual prompts" and "back-end validity adjudication" is adopted. The front end displays the end effector speed through a dual-dial or dual-bar speedometer. One dial displays the linear velocity percentage (range 0% to 100%), and the other displays the angular velocity percentage, which can be indicated by green, yellow, and red color partitions. The back end uses the Jacobian matrix relationship to convert the end effector linear velocity and angular velocity into estimated joint angular velocities. The conversion method can be the damped pseudo-inverse method. Alternatively, it can be obtained through the joint angle difference method, and then the estimated angular velocity of each joint is compared with the upper limit of the single joint velocity set in the hardware specifications of the robotic arm.

[0089] During data acquisition, the speedometer displays the speed status in real time, reminding the data acquisition operator to control the operating speed within a reasonable range. For example, when the ratio of linear velocity or angular velocity enters the yellow zone (such as the 80% to 90% threshold), the system issues a prompt, guiding the data acquisition operator to slow down the operating speed in the corresponding direction; when any ratio enters the red zone (exceeding the 90% or 100% threshold), it is determined that the speed has exceeded the limit, the corresponding task segment fails to execute, the system discards this part of the data, or automatically triggers a speed reduction and cleaning process to remove the overspeed data segment.

[0090] In one embodiment of this application, when the color currently displayed on the speedometer is the target color, the end-effector speed of the robotic arm is corrected by time scaling in order to complete the cleaning of the valid data.

[0091] The target color on the speedometer indicates that the current speed requires data cleaning. For example, red indicates overspeeding. Data segments that are "overspeeding but reachable" are slowed down, rather than being discarded directly. Overspeeding data can be corrected by time scaling. This involves calculating the ratio of all joint angular velocities in the frame to the corresponding joint velocity upper limit, and taking the maximum value as the time scaling factor. :

[0092]

[0093] And the time interval of this data segment Enlarged to This is equivalent to adjusting the overall velocity of motion according to... This reduces the speed by a factor of two, thereby bringing the speed of all joints down to within a safe threshold.

[0094] In addition, based on the key nodes of the original trajectory, the time axis can be replanned or interpolation can be performed to adjust the joint changes between adjacent frames to meet the speed limit of the robotic arm, while preserving the original motion trajectory features to the greatest extent.

[0095] In one embodiment of this application, the current frame data of the collected robotic arm data is obtained, and the dynamic reachable space corresponding to the current frame data is obtained; in a preset number of subsequent frame data, it is determined whether there is frame data located in the dynamic reachable space. If not, the collected data is discarded; if so, the frame that meets the condition closest to the current frame is marked as the target frame, all frame data from the current frame to the previous frame of the target frame are discarded, and the target frame is used as the new current frame, until the processing of all frame data in the collected data is completed.

[0096] For a completed UMI data set, processing can also be performed based on the trajectory of the dynamic reachability space. The current frame data of the collected data is read, and the dynamic reachability space of the robotic arm corresponding to that frame, i.e., the reachability ellipsoid range, is calculated in real time. Within a preset range of subsequent frames (sliding window), it is searched to see if there are any valid frames that fall back into the reachability space of the current frame. If no valid frame is found within the window, it means that the subsequent trajectory has completely deviated from the robotic arm's execution capability, and the entire data is discarded. If a valid frame is found, the nearest valid frame to the current frame is marked as the target frame, and all data from the current frame to the frame before the target frame is discarded (i.e., unreachable intermediate segments are removed). The target frame is then set as the new current frame for further processing. Finally, invalid segments in the trajectory are removed, and the valid segments are reassembled to generate a continuous and executable new trajectory.

[0097] Figure 3This document illustrates a flowchart of the cleaning process for collected UMI data in one embodiment of this application. For a single input UMI data entry, the frame index i=1 is initialized, and the hyperparameter n (the number of future frames to be verified) is set. For the i-th frame, the executable ellipsoid (or its 6-dimensional orientation margin index) corresponding to that frame is calculated. Within the range from the i-th frame to the (i+n)-th frame, it is searched for a frame that is closest to the i-th frame and whose end displacement / velocity requirement falls within the allowable range of the ellipsoid; this frame is denoted as the (i+k)-th frame. If no frame meeting the conditions exists within this range, the UMI data entry is discarded. If a frame meeting the conditions is found (i+k), the data from the i-th frame to the (i+k-1)-th frame is discarded, the frame index is updated to i=i+k, and the executable ellipsoid for the new index frame is calculated, repeating the above filtering process.

[0098] It should be noted that the data cleaning method in this application is not only applicable to robotic arm data generated by the data acquisition method proposed in this application, but can also be widely applied to robotic arm motion data from other sources. In practical applications, if the robotic arm configuration assumed during data acquisition differs from the robotic arm configuration used in the subsequent actual task execution (i.e., heterogeneous robotic arm scenario), the dynamic reachability space (i.e., reachability ellipsoid) corresponding to the actual execution arm configuration parameters is uniformly calculated. Then, following the same frame-by-frame verification process described above, data that falls within the allowable range of the execution arm ellipsoid throughout the entire process is selected, thereby ensuring the data's usability for migration between different hardware platforms.

[0099] In one embodiment of this application, the data acquisition process of the robotic arm is as follows: the data acquisition operator completes a preset operation task by holding the gripper, and the host industrial control computer synchronously records the end effector pose (EEF) data and video information collected by each camera. The UMI data acquisition system hardware may specifically include a triangulation positioner, a host industrial control computer, grippers, and vision sensors, etc.

[0100] The triangulation unit tracks the gripper's pose (position and orientation) in 3D space in real time, outputting key parameters such as the 6-DOF coordinates (X, Y, Z, Roll, Pitch, Yaw) of the end effector. The host computer serves as the system control center, receiving and processing all sensor data, coordinating the synchronous operation of various components, and managing data storage. The gripper performs grasping and various manipulation tasks. Two grippers can be used, each integrating multiple sensor modules, such as one stereo camera and one fisheye camera. The stereo camera provides stereo vision information and can be applied to depth perception, target recognition, and other scenarios. The fisheye camera is used for wide-field-of-view environmental monitoring, enabling panoramic recording of the surrounding environment.

[0101] Building upon this foundation, additional visual sensors can be added, including a first-person view camera and a third-person view camera. The first-person view camera simulates the operator's perspective and can be mounted on the operator's helmet or glasses; the third-person view camera records the overall scene and can be mounted on the top or side of the work area. During data acquisition, the host industrial computer uses a triangulator to read the relative 3D spatial pose of the grippers in real time and simultaneously acquires and records image data from the integrated cameras on each gripper and the additional cameras to collect complete task data samples.

[0102] The proposed solution utilizes the inverse kinematics (IK) space constraints of the robotic arm to achieve standardized and automated cleaning of UMI data acquisition. During the data acquisition phase, the IK executable ellipsoid of the robotic arm is calculated in real time and converted into a visual progress bar on the screen. By observing the progress bar feedback, the operator perceives the distance between the current action and the kinematic boundary of the robotic arm, thus consciously limiting their behavior within the executable space and fundamentally avoiding the generation of invalid data. During the data cleaning phase, real-time filtering ensures that the acquired data conforms to the dynamic execution standards. For old data acquired without using visualization standards, or data where the acquisition / execution space has changed significantly, the system automatically identifies and retains valid segments that conform to IK constraints, removing contaminated data and significantly improving the quality of the dataset.

[0103] Currently, data acquisition in the field of embodied intelligence relies on expensive robotic arm hardware, resulting in high construction costs. This solution fully reuses existing UMI wearable acquisition devices, significantly reducing hardware investment while ensuring the validity of the acquired data for the target robotic arm through inverse constraint decomposition technology. It can also evolve towards AIGC technology in the future, combining hand-eye calibration and video rendering technologies to directly transform human hand operations into robotic gripper movements, thereby completely eliminating reliance on physical robots and achieving a pure vision paradigm where the entire data acquisition process can be completed using only a head-mounted camera.

[0104] Figure 4 The diagram shows a schematic frame of an electronic device according to another aspect of this application, the electronic device including at least a processor 401 and a memory 402.

[0105] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0106] Memory 402 may include one or more computer-readable storage media, which may be non-transitory. Memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 402 are used to store at least one instruction, which is executed by processor 401 to implement a robotic arm data processing method provided in the method embodiments of this application.

[0107] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuits, touch displays, audio circuits, and power supplies.

[0108] Of course, the electronic device may also include fewer or more components, and this embodiment does not limit this.

[0109] This application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which can be executed by a processor to implement a robotic arm data processing method as described above.

[0110] When a robotic arm's data processing method is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0111] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.

[0112] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0113] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0114] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0115] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0116] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

Claims

1. A method for processing data with a robotic arm, characterized in that, The method includes: The current pose and configuration information of the robotic arm are obtained, and the dynamic reachable space of the robotic arm is determined based on the current pose and configuration information. The dynamic reachable space of the robotic arm is mapped onto the user interface and displayed as a visual graphic. The visual graphic includes a progress bar and a speed code table to display the reachability progress of the dynamic reachable space. The speed code table includes a linear speed code table and an angular speed code table. If the achievable progress is within the target area, then drive the robotic arm to continue performing data collection operations within the target area; If the achievable progress is not within the target area, the collected data that is not within the target area will be cleaned. The step of cleaning the collected data that is not within the specified compliance area includes: Extract the target pose information of each frame of data from the collected data, and determine the joint angle parameters based on the target pose information; The pose error is determined based on the joint angle parameters, and the collected data is filtered based on the pose error and a preset error threshold to obtain valid data. The end effector speed of the robotic arm is determined based on the joint angle parameters, and the speed code table is determined based on the end effector speed of the robotic arm, wherein the speed code table is used to display the speed percentage by color partitioning. The valid data is cleaned according to the color currently displayed on the speedometer. The step of determining the end effector speed of the robotic arm based on the joint angle parameters, and determining the speedometer based on the end effector speed of the robotic arm, includes: The end effector velocity of the robotic arm is determined based on the joint angle parameters, and the linear velocity component and angular velocity component are determined based on the components of the end effector velocity of the robotic arm in different spatial dimensions. A linear velocity ratio is generated based on the linear velocity components and a preset linear velocity threshold, and the linear velocity ratio is mapped to a linear velocity code table. An angular velocity ratio is generated based on the angular velocity components and a preset angular velocity threshold, and the angular velocity ratio is mapped to an angular velocity code table.

2. The method according to claim 1, characterized in that, Determining the dynamic reachability space of the robotic arm based on the current pose information and configuration information includes: The joint state parameters of the robotic arm are determined based on the current pose information and configuration information. The Jacobian matrix is ​​determined based on the joint state parameters to establish the correlation between the joint velocity and the end effector velocity of the robotic arm. Set the constraints on the joint velocity, and determine the parameter set of the end effector motion based on the Jacobian matrix and the constraints. Based on the Jacobian matrix, the set of parameters of the end effector motion is transformed into a corresponding geometric space to determine the dynamic reachability space of the robotic arm.

3. The method according to claim 1, characterized in that, The process of mapping the dynamic reachability space of the robotic arm onto the user interface for visual graphical display includes: Determine multiple preset directions in the dynamic reachability space and the maximum reachability index in each preset direction; The maximum reachable motion index is normalized to obtain a standard reachable capacity index for each preset direction, and the standard reachable capacity index is mapped onto the user interface for visual display.

4. The method according to claim 1, characterized in that, The step of cleaning the valid data according to the color currently displayed on the speedometer includes: When the color currently displayed on the speedometer is the target color, the end effector speed of the robotic arm is corrected by time scaling to complete the cleaning of the valid data.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the current frame data of the collected robotic arm data, and obtain the dynamic reachable space corresponding to the current frame data; In a preset number of subsequent frame data, it is determined whether there is any frame data located within the dynamic reachable space; if not, the collected data is discarded. If so, mark the frame that meets the condition closest to the current frame as the target frame, discard all frame data from the current frame to the frame before the target frame, and use the target frame as the new current frame, until the processing of all frame data in the collected data is completed.

6. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the operations of the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, The computer-readable instructions can be executed by a processor to implement the method as described in any one of claims 1 to 5.

Images