A teaching robot operation track three-dimensional evaluation method, system, device and storage medium

Abstract

The present application relates to teaching robot practice evaluation related technology, provide a kind of teaching robot operation trajectory three-dimensional evaluation method, system, equipment and storage medium.The method obtains the image sequence of teaching robot in the process of executing practice task collected by fixed external camera, identifies and continuously tracks end effector, tool center point or preset key position, obtains two-dimensional operation trajectory information;According to the camera internal parameter and external parameter, the two-dimensional trajectory point is mapped into space ray, and the three-dimensional operation trajectory under world coordinate system is solved in combination with constraint optimization model;Again, according to key event, key action node or preset space region, action stage is divided and time sequence is aligned, and matching score is generated for each action stage and output trajectory score result.The scheme is suitable for practice task process evaluation in teaching robot platform.

Description

Technical Field

[0001] This invention relates to the field of practical evaluation technology for teaching robots, specifically to a three-dimensional evaluation method, system, device, and storage medium for the operation trajectory of a teaching robot. Background Technology

[0002] Educational robot platforms have been widely used in teaching activities such as robotic arm control, grasping and handling, trajectory planning, assembly demonstration, and comprehensive practical training. In these platforms, the evaluation object is usually not simply the production result, but rather the operational quality of students or trainees during the execution of practical tasks. When evaluating the completion of practical tasks using educational robots, teachers typically focus not only on whether the task was completed, but also on whether the trajectory was standardized, whether key movements were accurate, whether the movement was smooth, whether the transitions between different movement stages were reasonable, and whether the specific reasons for lost points can be clearly identified.

[0003] Unlike conventional robot evaluation methods for industrial production or general operational scenarios, evaluations in educational robot platforms typically emphasize process quantification, standardized approaches, and interpretable teaching feedback. Conventional robot evaluation methods often focus on production cycle time, processing quality, endpoint position error, repeatability, path following error, or controller internal state parameters, and rely heavily on robot controller logs, joint encoder data, end-effector sensor data, or specialized process detection results. While these evaluation methods are suitable for industrial production or general robot operation monitoring, they are difficult to directly apply to the operational process evaluation needs of educational robot platforms, especially in terms of generating unified, intuitive, and interpretable evaluation results across educational platforms of different brands, configurations, or with varying levels of interface openness.

[0004] Current teaching evaluation methods mainly rely on teachers' on-site observation and post-event manual scoring. On the one hand, teachers can intuitively judge whether the movements are standardized, but this method is greatly influenced by experience and it is not easy to form a unified quantitative standard. On the other hand, if only the completion of the task is recorded, it is difficult to distinguish different situations such as completion but obvious trajectory deviation, completion but large process jitter, and completion but inaccurate hitting of key movements. It is also difficult to provide learners with feedback on which specific movement stage and what kind of spatial deviation caused the score to drop.

[0005] In existing technologies, most common visual acquisition devices acquire two-dimensional image data. The trajectory points in a two-dimensional image correspond only to pixel positions on the image plane, while the actual movement of the end effector of a teaching robot occurs in a three-dimensional workspace. When the robotic arm performs actions such as lifting, pressing down, obstacle avoidance, crossing, or translation at different heights, similar two-dimensional projections may correspond to significantly different three-dimensional trajectories. Therefore, if evaluation is based solely on two-dimensional trajectories, it is easy to fail to adequately assess spatial offset, spatial pullback, and the spatial hit rate of critical nodes, thus making it difficult to meet the requirements of teaching robot platforms for precise evaluation of the operational process.

[0006] On the other hand, existing 2D-to-3D conversion schemes are mostly geared towards general image 3D reconstruction, binocular measurement, target point cloud modeling, or path planning scenarios. Existing conventional robot evaluation methods also largely focus on industrial process execution results, controller internal data, or general motion errors. They typically fail to address the unique characteristics of teaching robot practical tasks, such as fixed external visual observation, known robotic arm structure, clearly defined task space boundaries, predefined key action positions, constructable standard teaching trajectories, and the need for scoring results to serve teaching feedback. Directly employing general reconstruction or robot evaluation methods can easily lead to problems such as significant depth ambiguity, strong local jitter, unclear stage divisions, insufficient interpretability of evaluation results, and a disconnect from teaching guidance.

[0007] Furthermore, existing trajectory scoring schemes are mostly based on two-dimensional path differences or general motion errors, lacking a complete technical solution for dividing three-dimensional operational trajectories and preset standard three-dimensional trajectories according to task action stages, aligning them temporally, matching them across stages, and outputting stage scores and overall scores. In teaching scenarios, teachers would prefer to know exactly which action stage and what spatial deviation caused the score drop in order to provide targeted guidance. However, conventional robot evaluation results typically cannot directly provide this type of structured information for teaching feedback.

[0008] Therefore, a technical solution is needed for teaching robot platforms that can stably map the two-dimensional operation trajectory of the teaching robot in the process of performing practical tasks into a three-dimensional operation trajectory in the world coordinate system under fixed external vision conditions, and perform action stage division, temporal alignment and matching scoring with the preset standard three-dimensional trajectory, so as to realize objective, interpretable and feedback-able trajectory evaluation suitable for teaching scenarios. Summary of the Invention

[0009] The purpose of this invention is to provide a three-dimensional evaluation method, system, device, and storage medium for the operation trajectory of teaching robots. This addresses the problem that existing conventional robot evaluation methods are mainly geared towards industrial production, general operation results, or internal controller states, and are difficult to directly apply to the application needs of fixed external visual observation, process trajectory quantification, staged comparative scoring, and interpretability of teaching feedback in teaching robot platforms. Furthermore, this invention solves the problems in existing technologies, such as the inability of two-dimensional visual trajectories to directly reflect three-dimensional spatial motion characteristics, the lack of teaching robot scene constraints in two-dimensional to three-dimensional mapping, the lack of action stage division and temporal alignment mechanisms in standard three-dimensional trajectory matching, and the difficulty in directly serving teaching guidance with trajectory scoring results.

[0010] In a first aspect, the present invention provides a three-dimensional evaluation method for the operation trajectory of a teaching robot, comprising: acquiring an image sequence of a teaching robot performing a practical task captured by a fixed external camera; identifying and continuously tracking the end effector, tool center point, or preset key parts of the teaching robot to obtain two-dimensional operation trajectory information; mapping each two-dimensional trajectory point in the two-dimensional operation trajectory information to a spatial ray according to the intrinsic and extrinsic parameters of the camera, and constructing a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system using the spatial depth parameter corresponding to each trajectory point as an optimization variable; solving for the spatial position of each trajectory point in the world coordinate system based on the constrained optimization model to generate three-dimensional operation trajectory information; dividing the three-dimensional operation trajectory information and preset standard three-dimensional trajectory information into action stages and aligning them temporally according to the key events, key action nodes, or preset spatial regions corresponding to the practical task; generating a stage matching score for each action stage based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency, and spatial smoothness deviation; generating a trajectory scoring result based on the matching score of each action stage; and outputting the trajectory scoring result.

[0011] In a preferred embodiment, the intrinsic and extrinsic parameters of the camera include at least the camera intrinsic parameter matrix, rotation matrix, and translation vector; the constraint optimization model includes at least three of the following: projection consistency constraint, robotic arm reachable space constraint, task area constraint, key action position constraint, and trajectory smoothing constraint.

[0012] In a preferred embodiment, the step of mapping each two-dimensional trajectory point in the two-dimensional operation trajectory information to a spatial ray based on the camera's intrinsic and extrinsic parameters, and constructing a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system using the spatial depth parameter corresponding to each trajectory point as an optimization variable, and solving for the spatial position of each trajectory point in the world coordinate system based on the constrained optimization model to generate three-dimensional operation trajectory information, includes: determining the spatial ray direction vector in the camera's normalized plane based on the two-dimensional trajectory points and the camera's intrinsic parameter matrix; establishing the correspondence between three-dimensional points in the camera coordinate system and the spatial ray direction vector and spatial depth parameter; transforming the three-dimensional points in the camera coordinate system to the world coordinate system based on the camera's extrinsic parameters; constructing a constrained optimization model for describing the mapping relationship between two-dimensional trajectory points and three-dimensional trajectory points in the world coordinate system, and solving for the spatial depth parameter corresponding to each trajectory point based on the constrained optimization model to generate the three-dimensional operation trajectory information.

[0013] In a preferred embodiment, the step of dividing the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information into action stages and aligning them temporally based on key events, key action nodes, or preset spatial regions corresponding to the practical task, and generating a trajectory scoring result based on the matching score of each action stage, includes: dividing the three-dimensional operation trajectory obtained by the constraint optimization model and the preset standard three-dimensional trajectory into action stages based on key events, key action nodes, or preset spatial regions corresponding to the practical task; aligning each action stage temporally, and generating a trajectory matching score for each stage based on the matching status of each action stage; and generating the trajectory scoring result based on the trajectory matching score of each stage.

[0014] In a preferred embodiment, the matching status of each action stage includes at least two or more of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency, and spatial smoothness deviation.

[0015] Preferably, the first The matching score for each action phase is determined as follows:

[0016]

[0017] in, Indicates the first Matching score for each action phase Indicators representing spatial shape consistency This indicates that key action nodes have been hit by the indicator. Indicators representing the consistency of three-dimensional motion trends This represents the spatial smoothing deviation penalty index. , , , This represents the weighting coefficient of the corresponding indicator.

[0018] In a preferred embodiment, the trajectory scoring result includes a comprehensive three-dimensional trajectory score, which is determined as follows:

[0019]

[0020] in, This represents the comprehensive three-dimensional trajectory score. Indicates the number of action stages. Indicates the first The scoring weights corresponding to each action phase Indicates the first Matching score for each action phase.

[0021] In a preferred embodiment, when the confidence level of the two-dimensional trajectory point detection at a certain moment is lower than a preset threshold or a short-term missing value occurs, a weight coefficient corresponding to the detection confidence level is introduced into the projection consistency constraint term, and the spatial depth parameter at that moment is initialized and solved using the three-dimensional trajectory point at the previous moment, the corresponding task area constraint, and the trajectory continuity constraint.

[0022] Secondly, the present invention also provides a three-dimensional evaluation system for the operation trajectory of a teaching robot, comprising: a data acquisition module, used to acquire image sequences of the teaching robot performing practical tasks captured by a fixed external camera, and to identify and continuously track the end effector, tool center point, or preset key parts of the teaching robot to obtain two-dimensional operation trajectory information; and a three-dimensional mapping module, used to map each two-dimensional trajectory point in the two-dimensional operation trajectory information into a spatial ray according to the intrinsic and extrinsic parameters of the camera, and to construct a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system using the spatial depth parameters corresponding to each trajectory point as optimization variables. The system calculates the spatial position of each trajectory point in the world coordinate system based on the constrained optimization model to generate three-dimensional operation trajectory information. The trajectory scoring module is used to divide the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information into action stages and align them temporally according to the key events, key action nodes or preset spatial regions corresponding to the practical task. For each action stage, it generates a stage matching score based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency and spatial smoothness deviation. The system generates a trajectory scoring result based on the matching score of each action stage. The output module is used to output the trajectory scoring result.

[0023] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein when the processor executes the computer program, it implements the method described in any of the above embodiments.

[0024] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the method described in any of the above embodiments.

[0025] Compared to existing technologies, this invention addresses the challenges of teaching robot platforms, which typically rely on fixed external vision devices for process observation, exhibit inconsistent levels of openness in the internal interfaces of different teaching devices, and struggle to uniformly evaluate based on controller internal data. It acquires two-dimensional operational trajectories using fixed external vision and solves for three-dimensional operational trajectories in the world coordinate system by constructing an optimization model incorporating various scenario constraints. This enables the identification of operational results with similar two-dimensional projections but significantly different three-dimensional paths without relying on the robot's internal closed-loop control data, thereby improving the accuracy of spatial evaluation and cross-platform applicability in teaching robot platforms. Furthermore, considering the characteristics of teaching robot practical tasks where the robotic arm structure is known, the work area is defined, key action positions can be predefined, and standard teaching trajectories can be constructed, this invention introduces constraints such as camera intrinsic and extrinsic parameters, robotic arm reachable space, task area, key action positions, and trajectory smoothing. This effectively alleviates depth ambiguity in the two-dimensional to three-dimensional solution process, improving the stability and task consistency of the three-dimensional trajectory solution. Furthermore, this invention divides action stages and aligns their timing based on key events, key action nodes, or preset spatial regions, and performs structured matching scoring for each stage. This clearly identifies the sources of spatial deviation and reasons for point deductions in different action stages, thereby improving the interpretability, feedback capability, and pedagogical guidance value of the scoring results. Moreover, in complex teaching and training environments with multiple operators, partial occlusion, reflections, lighting variations, and short-term detection instability, this invention can still achieve stable solutions by utilizing confidence-weighted projection consistency constraints, task region constraints, and trajectory continuity constraints, even when detection confidence decreases or is temporarily lost. This enhances the robustness and continuous evaluation capability of the teaching robot platform in complex teaching and training environments, making the evaluation results more stable and suitable as a basis for teaching feedback and practical evaluation. Attached Figure Description

[0026] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0027] Figure 1 The overall flowchart of the three-dimensional evaluation method for the operation trajectory of the present invention is shown;

[0028] Figure 2 This diagram illustrates how the present invention maps two-dimensional trajectory points to spatial rays and determines their spatial positions in the world coordinate system.

[0029] Figure 3 A schematic diagram of the constrained optimization model of the present invention is shown;

[0030] Figure 4 This diagram illustrates the three-dimensional trajectory motion phase matching and scoring method of the present invention.

[0031] Figure 5 A structural block diagram of the three-dimensional evaluation system for operational trajectories of the present invention is shown. Detailed Implementation

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] To facilitate understanding of this invention, the main terms used herein will be explained first. The world coordinate system mentioned herein refers to a reference coordinate system established based on camera calibration parameters, robot base installation parameters, workspace parameters, or combinations thereof, used to uniformly describe the relationships between various three-dimensional positions in the workspace of the teaching robot. If multiple local coordinate systems exist in the system, they can be unified to the same world coordinate system for solving and scoring.

[0034] The two-dimensional operation trajectory information mentioned in this paper refers to the image sequence acquired by a fixed external camera during the execution of practical tasks by a teaching robot, and the image planar temporal trajectory obtained after identifying and continuously tracking the end effector, tool center point, or preset key parts. The three-dimensional operation trajectory information mentioned in this paper refers to the temporal trajectory in the world coordinate system obtained by solving the two-dimensional operation trajectory information, camera intrinsic and extrinsic parameters, and constraint optimization model. The preset standard three-dimensional trajectory information mentioned in this paper refers to the reference trajectory formed based on the teacher's demonstration trajectory, excellent sample trajectories, or predefined standard action templates, used for phased matching and scoring with the three-dimensional operation trajectory to be evaluated.

[0035] The main symbols in this article have the following meanings: Indicates the first Two-dimensional trajectory points at any given moment; Indicates the first The three-dimensional trajectory points at any given moment; Indicates the first The spatial ray direction vector corresponding to the two-dimensional trajectory point at time moment; Indicates the first The spatial depth parameter corresponding to the given time; Represents the camera intrinsic parameter matrix; and These represent the rotation matrix and translation vector in the camera's extrinsic parameters, respectively. This represents the overall optimization objective; Indicates the projection consistency constraint term; This indicates the reachable space constraints for the robotic arm; Indicates task area constraints; Indicates the position constraints of key actions; This represents the trajectory smoothing constraint term; Indicates the first Match score for each action phase; This represents the comprehensive three-dimensional trajectory score.

[0036] See also Figures 1 to 5 As shown, the present invention provides a method, system, device and storage medium for three-dimensional evaluation of the operation trajectory of a teaching robot.

[0037] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0038] Step S101: Obtain the image sequence of the teaching robot performing the practical task captured by the fixed external camera, identify and continuously track the end effector, tool center point or preset key parts of the teaching robot, and obtain two-dimensional operation trajectory information;

[0039] Step S102: Based on the camera's intrinsic and extrinsic parameters, map each two-dimensional trajectory point in the two-dimensional operation trajectory information to a spatial ray, and use the spatial depth parameter corresponding to each trajectory point as an optimization variable to construct a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system. Based on the constrained optimization model, solve for the spatial position of each trajectory point in the world coordinate system to generate three-dimensional operation trajectory information.

[0040] Step S103: Based on the key events, key action nodes or preset spatial regions corresponding to the practical task, divide the three-dimensional operation trajectory information and preset standard three-dimensional trajectory information into action stages and align them in time sequence. For each action stage, generate a stage matching score based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency and spatial smoothness deviation. Generate a trajectory scoring result based on the matching score of each action stage.

[0041] Step S104: Output the trajectory score results.

[0042] Specifically, step S101 addresses the problem of acquiring two-dimensional trajectory data; step S102 addresses the problem of recovering a three-dimensional trajectory in the world coordinate system from two-dimensional observation under fixed external visual conditions; and step S103 addresses the problem of how to perform action stage division, temporal alignment, and quantitative evaluation based on a standard three-dimensional trajectory. Therefore, the overall technical logic of this invention is: image sequence acquisition and two-dimensional trajectory extraction, three-dimensional spatial solution, and staged standard trajectory matching and scoring.

[0043] As can be seen from the above description, the above embodiments of the present invention, by placing two-dimensional visual trajectory acquisition, three-dimensional trajectory solving and standard trajectory stage scoring into a complete technology chain, can not only obtain three-dimensional operation trajectories that can be used for teaching evaluation, but also directly output interpretable stage scores and comprehensive scores for teaching scenarios, forming a complete solution that can be engineered and implemented.

[0044] In one embodiment, the two-dimensional operation trajectory information is obtained by identifying and continuously tracking the end effector, tool center point, or preset key parts of the teaching robot. Possible methods include: setting color markers, QR code markers, or geometric markers on the end effector and obtaining the two-dimensional position through image recognition; or using target detection and key point detection models to directly locate natural feature points such as the gripper center and tool tip; or using template matching, contour tracking, optical flow tracking, etc., to obtain a continuous two-dimensional trajectory.

[0045] Preferably, to improve stability, the original video frames can first undergo distortion correction, background suppression, brightness normalization, and region of interest cropping before target recognition and continuous tracking. Let the first frame be... The two-dimensional trajectory points obtained in the frame are Then, the two-dimensional trajectory points at all times form two-dimensional operation trajectory information in chronological order. .

[0046] As can be seen from the above description, the above embodiments of the present invention clearly define the acquisition object and acquisition process of the two-dimensional operation trajectory, providing a stable and repeatable data foundation for subsequent three-dimensional trajectory solving, and retaining compatibility with different vision acquisition schemes.

[0047] In one embodiment, the two-dimensional trajectory points are first mapped to spatial rays through camera imaging relationships. Let the camera intrinsic parameter matrix be... Two-dimensional trajectory points are Then its direction vector in the camera normalized plane It can be represented as:

[0048]

[0049] If, in the camera coordinate system, the camera optical center is taken as the origin, then the corresponding 3D point lies on the ray. Above can be represented as:

[0050]

[0051] in, Represents a 3D point in the camera coordinate system. This represents the depth scale parameter to be solved. If the camera extrinsic parameters are rotation matrices... Translation vector Then a three-dimensional point in the world coordinate system It can be represented as:

[0052]

[0053] Substituting the above equation, we get:

[0054]

[0055] Therefore, for each two-dimensional trajectory point Its world coordinate system position is determined by the depth scale parameter. The solution is uniquely determined. Therefore, one of the key aspects of this invention lies in combining camera parameters, structural constraints, and task constraints in the teaching robot scenario to stably solve for the time-varying parameters at each moment. .

[0056] As can be seen from the above description, the above-described embodiments of the present invention can convert two-dimensional visual observation into a three-dimensional trajectory representation in a unified coordinate system by first mapping two-dimensional trajectory points to spatial rays and then solving for spatial positions in the world coordinate system, thus providing a basis for subsequently introducing various constraints and performing standard trajectory scoring.

[0057] In one embodiment, the camera's intrinsic and extrinsic parameters include at least a camera intrinsic parameter matrix. Rotation matrix Translation vector The constrained optimization model includes at least three of the following constraints: projection consistency constraint, robotic arm reachable space constraint, task area constraint, key action position constraint, and trajectory smoothing constraint.

[0058] Among them, camera intrinsic and extrinsic parameters are used to establish the geometric relationship between two-dimensional image coordinates, camera coordinate system and world coordinate system; robotic arm reachability space constraint is used to restrict three-dimensional trajectory points from falling into areas inaccessible to the robotic arm; task area constraint is used to restrict the obtained three-dimensional trajectory to conform to the work space logic corresponding to the specific task; key action position constraint is used to anchor trajectory points at critical moments; trajectory smoothing constraint is used to suppress three-dimensional trajectory jumps caused by two-dimensional observation noise or local occlusion.

[0059] As can be seen from the above description, the above-described embodiments of the present invention, by unifying the geometric, structural, task, and process information in the teaching robot scenario into solution constraints, not only narrow the feasible solution space from two-dimensional to three-dimensional solution, but also improve the physical rationality and task consistency of the three-dimensional trajectory results.

[0060] In one embodiment, based on the camera's intrinsic and extrinsic parameters, each two-dimensional trajectory point in the two-dimensional operation trajectory information is mapped to a spatial ray. Using the spatial depth parameter corresponding to each trajectory point as an optimization variable, a constrained optimization model is constructed for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system. Based on the constrained optimization model, the spatial position of each trajectory point in the world coordinate system is obtained to generate the three-dimensional operation trajectory information. This includes: determining the spatial ray direction vector in the camera's normalized plane based on the two-dimensional trajectory points and the camera's intrinsic parameter matrix; establishing the correspondence between the three-dimensional points in the camera coordinate system and the spatial ray direction vector and spatial depth parameter; transforming the three-dimensional points in the camera coordinate system to the world coordinate system based on the camera's extrinsic parameters; constructing a constrained optimization model to describe the mapping relationship between two-dimensional trajectory points and three-dimensional trajectory points in the world coordinate system; and obtaining the spatial depth parameter corresponding to each trajectory point based on the constrained optimization model to generate the three-dimensional operation trajectory information.

[0061] In another embodiment, the constrained optimization model is constructed as follows:

[0062]

[0063] in, This represents the overall optimization objective. This represents the projection consistency constraint term. This indicates the reachable space constraint term for the robotic arm. Indicates task region constraints. This indicates the position constraints of key actions. This represents the trajectory smoothing constraint term. to This represents the weight coefficient of the corresponding constraint term.

[0064] In another embodiment, the projection consistency constraint is determined as follows:

[0065]

[0066] in, Indicates the first The three-dimensional trajectory points at time [time]. Denotes the two-dimensional trajectory point at time t. This represents the projection function from three-dimensional space to a two-dimensional image plane. This represents the total number of trajectory points.

[0067] In another embodiment, the trajectory smoothing constraint term is determined as follows:

[0068]

[0069] The first term is used to constrain the positional continuity of adjacent trajectory points, and the second term is used to constrain the smoothness of trajectory changes.

[0070] 1. Overall Optimization Objective

[0071] Let the set of three-dimensional trajectory points at all times be . The overall optimization objective is defined as:

[0072]

[0073] in, to These are non-negative weighting coefficients used to balance the impact of various constraint terms on the final solution. This overall optimization objective does not directly participate in subsequent scoring, but is used to solve for the 3D operational trajectory to be evaluated; the 3D trajectory points at each time step are obtained by solving for E. The three-dimensional operational trajectory to be evaluated is constructed in chronological order. Subsequent ratings will be based on this. conduct.

[0074] 2. Projection Consistency Constraints

[0075] The projection consistency constraint term is used to ensure that the obtained 3D trajectory points, after being reprojected onto the 2D image plane, remain as close as possible to the original 2D observations. Its expression is:

[0076]

[0077] in, This is the projection function. The smaller this term is, the higher the consistency between the 3D trajectory and the original 2D observation.

[0078] 3. Reachability constraints of the robotic arm

[0079] The reachable space constraint term for the robotic arm is used to limit the spatial area that the three-dimensional trajectory point can reach at the end of the robotic arm. Inside can be represented as:

[0080]

[0081] in, This represents the distance from a point to a region. When a point on a trajectory is inside the reachable region, the distance is 0; when it is outside, the distance is the corresponding minimum external distance.

[0082] 4. Task Region Constraints

[0083] The task region constraint is used to restrict the 3D trajectory points to match the task region corresponding to the current action phase. If the... The moment belongs to the action phase ,but:

[0084]

[0085] in, Indicates the first The allowed spatial area for the action phase to which the moment belongs. For example, during the grasping phase, the trajectory point should be located near the grasping area, and during the placement phase, the trajectory point should be located near the placement area. Action Phase Label The stage index can be predetermined by task scripts, PLC events, key action nodes, or predefined spatial regions, and used as input for the stage constraints of the task region. During the scoring stage, stage-based comparisons can be performed based on the same stage labels or their alignment results.

[0086] 5. Position constraints for critical actions

[0087] Let the set of key action moments be . The corresponding three-dimensional position of the target is Then the key action position constraint can be expressed as:

[0088]

[0089] This feature is used to anchor trajectory points at critical moments such as grabbing, placing, starting, and returning to zero, in order to ensure the reliability of the spatial position of critical actions.

[0090] 6. Trajectory smoothing constraint

[0091] The trajectory smoothing constraint is used to suppress local jitter and unreasonable jumps, and its expression is:

[0092]

[0093] The first term is used to constrain the positional continuity of adjacent trajectory points, and the second term is a second-order difference term used to constrain the smoothness of trajectory changes.

[0094] 7. Solution Method

[0095] In the specific solution, the depth scale parameter set can be used. As an optimization variable, Substitute into the overall optimization objective In this process, the Gauss-Newton method, the Levenberg-Marquardt method, or other nonlinear optimization methods are used for iterative solutions to obtain the optimal set of scale parameters. This leads to the set of three-dimensional trajectory points. .

[0096] As can be seen from the above description, the above-described embodiments of the present invention can significantly reduce depth ambiguity and trajectory jitter in the two-dimensional to three-dimensional mapping by solving multiple constraints jointly, rather than simply relying on a certain type of depth information, and obtain a three-dimensional operation trajectory that is more in line with the requirements of the robotic arm structure and task space.

[0097] In one embodiment, when the confidence level of the detection of two-dimensional trajectory points at a certain moment is lower than a preset threshold or a short-term missing value occurs, a weight coefficient corresponding to the detection confidence level is introduced into the projection consistency constraint term, and the spatial depth parameter at that moment is initialized and solved using the three-dimensional trajectory points at the previous moment, the corresponding task area constraint, and the trajectory continuity constraint.

[0098] Specifically, when the confidence level of the two-dimensional trajectory point detection at a certain moment is lower than a preset threshold or a short-term missing value occurs, a weight coefficient corresponding to the detection confidence level is introduced into the projection consistency constraint term. This allows the projection consistency constraint term to be expressed as:

[0099]

[0100] in, The detection confidence of the corresponding two-dimensional trajectory point is positively correlated; the lower the detection confidence, the more likely the corresponding... The smaller the value, the better. At this point, the three-dimensional trajectory points from the previous moment can be used. Corresponding task area constraints and trajectory continuity constraints on the spatial depth parameters at the current moment Initialization and solution are performed. Preferably, when trajectory points only experience short-term absences, the initialization and solution can be performed. The initial position is projected onto the vicinity of the allowable spatial region corresponding to the current moment, and then optimized by combining trajectory continuity constraints and other constraint terms. It can be seen that the above scheme can improve the stability of 3D trajectory solving in complex teaching and training environments such as occlusion, reflection, and short-term loss.

[0101] In one alternative implementation, if the system employs a multi-camera deployment, then It can be expanded to the confidence weights corresponding to each viewpoint. The system can perform weighted fusion of the projection consistency constraint terms according to the observation quality of each viewpoint. However, regardless of whether a single camera or multiple cameras are used, the trajectory points in the two-dimensional image sequence are used as the basis for observation, and the three-dimensional trajectory is solved by using the task area constraint and trajectory continuity constraint.

[0102] In one embodiment, the step of dividing the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information into action stages and aligning them temporally based on key events, key action nodes, or preset spatial regions corresponding to the practical task, and generating a trajectory scoring result based on the matching score of each action stage, includes: dividing the three-dimensional operation trajectory obtained by the constraint optimization model and the preset standard three-dimensional trajectory into action stages based on key events, key action nodes, or preset spatial regions corresponding to the practical task; aligning each action stage temporally, and generating a trajectory matching score for each stage based on the matching status of each action stage; and generating the trajectory scoring result based on the trajectory matching score of each stage.

[0103] The criteria for dividing the action phase may include at least one of the following: task trigger events output by the robot controller or PLC, changes in the state of the end effector, the trajectory entering a preset spatial region, changes in the speed threshold, and the hit time of key action nodes. Timing alignment can be achieved using an alignment method based on key node anchoring, or by using a dynamic time warping method to align the trajectory to be evaluated with a standard trajectory within the same action phase, ensuring that the phase comparison remains comparable under different execution speeds or partial pauses.

[0104] In another embodiment, the preset standard three-dimensional trajectory information can be constructed as follows: collect the teacher's demonstration trajectory or multiple excellent sample trajectories, and resample each sample trajectory by performing coordinate system one, action stage division, time or arc length normalization; generate stage standard trajectories or stage standard trajectory bands for multiple sample trajectories within the same action stage by using dynamic time warping, mean fusion or envelope band fitting; and then stitch together the stage standard trajectories of each action stage according to the task execution order to obtain the preset standard three-dimensional trajectory information corresponding to the practical task.

[0105] Let the three-dimensional trajectory to be evaluated be:

[0106]

[0107] Let the preset standard three-dimensional trajectory be:

[0108]

[0109] After being divided into stages according to the action phase, they are represented as follows:

[0110]

[0111] in, This represents the number of action stages. The matching status of each action stage includes at least two or more of the following: spatial shape consistency, key action node hit rate, 3D motion trend consistency, and spatial smoothness deviation. In a preferred embodiment, the first... The matching score for each action phase is determined as follows:

[0112]

[0113] in, , , , These are the weighting coefficients.

[0114] 1. Spatial shape consistency index

[0115] Preferably, the trajectory segment to be evaluated is first... and standard trajectory segment Resampling by arc length normalization yields L three-dimensional sampling points:

[0116]

[0117]

[0118] The spatial shape consistency index can then be defined as:

[0119]

[0120] in, For the first The spatial normalized scale parameter for each action stage. This formula reflects the average deviation of the overall spatial morphology at that stage; the smaller the deviation, the closer the index is to 1.

[0121] 2. Key Action Node Hit Indicators

[0122] If the first Phase includes There are 1 key action nodes, and the key nodes corresponding to the trajectory to be evaluated and the standard trajectory are respectively denoted as _____. and The key action node hit index can then be defined as:

[0123]

[0124] The larger the value of this indicator, the more accurate the key action is in three-dimensional space.

[0125] 3. Three-dimensional motion trend consistency index

[0126] Suppose the trajectory segment to be evaluated and the standard trajectory segment are at the nth... The overall displacement vectors for each stage are as follows:

[0127]

[0128]

[0129] The consistency index of three-dimensional motion trends can then be defined as:

[0130]

[0131] This indicator reflects the degree of consistency between the overall three-dimensional motion direction and the standard direction.

[0132] 4. Spatial smoothing deviation penalty index

[0133] Preferably, the spatial smoothing deviation penalty index can be obtained from the three-dimensional second-order difference average of the trajectory segment to be evaluated:

[0134]

[0135] The larger this indicator is, the more obvious the local stuttering and jittering of the trajectory in three-dimensional space, and therefore it is used as a penalty item in the stage score.

[0136] 5. Comprehensive three-dimensional trajectory score

[0137] Obtain matching scores for each action phase. Then, the comprehensive three-dimensional trajectory score can be obtained by summing the results according to the stage weights:

[0138]

[0139] in, For the first The scoring weights corresponding to each stage should be optimized to meet the following criteria. and For tasks involving grasping and moving, the grasping and placement phases can be given greater weight; for tasks involving trajectory following, the turning and endpoint positioning phases can be given greater weight.

[0140] As can be seen from the above description, the above-described embodiments of the present invention, by establishing the scoring on a three-dimensional trajectory in the world coordinate system and dividing, aligning and structurally matching it according to the action stages, can avoid misjudgment caused by two-dimensional projection approximation, and at the same time can clarify the source of spatial deviation in different action stages, thereby improving the objectivity of the scoring and the interpretability of teaching.

[0141] To more fully illustrate the technical solution of this invention, a complete embodiment is given below in conjunction with a grasping, handling, and placing task. Suppose a teaching robot performs a task of moving a workpiece from the picking area to the placing area. The system uses a fixed external monocular camera to capture operation video, obtaining a total of T=80 sampling points of the two-dimensional operation trajectory of the end effector's center point in the image.

[0142] Step 1: 2D Trajectory Acquisition. A 2D trajectory point set is obtained through target detection and key point localization. .in, For the initial position, The nearby area corresponds to the material picking area. The corresponding grabbing action nearby Nearby corresponding transportation and turning points, The corresponding placement action nearby This is a pullback position.

[0143] Step 2: Ray mapping. Given the camera intrinsic parameter matrix. and external references For each two-dimensional trajectory point Calculate the normalized direction vector And establish three-dimensional points in the camera coordinate system. .in, The depth scale parameter to be solved.

[0144] Step 3: Establish spatial constraints. For this task, the constraint optimization model should at least include: the reachable space determined by the coordinates of the robot arm base and the length of the link. The task area consists of a material picking area, a transport area, and a placement area. ; Catch point and placement point The key action position constraints constitute the trajectory; and the trajectory smoothing constraints ensure that adjacent trajectory points are continuous and the trajectory changes are smooth.

[0145] Step 4: Establish the overall optimization objective E. This is done using all depth scale parameters. To optimize the variables, a total optimization objective is constructed based on the projection consistency constraint, the reachable space constraint of the robotic arm, the task area constraint, the position constraint of key actions, and the trajectory smoothing constraint. The optimal set of scale parameters was obtained by iteratively solving the problem using the Levenberg-Marquardt method. .

[0146] Step 5: Generate the 3D operation trajectory. Substitute return This yields 80 three-dimensional trajectory points in the world coordinate system. These points, arranged in chronological order, constitute the three-dimensional operational trajectory to be evaluated. .

[0147] Step Six: Action Phase Division and Timing Alignment. Based on the task logic, key task events output by the robot controller, and key action nodes, the sequence is... The motion is divided into four stages: approach, grasp, transport, and placement; a preset standard 3D trajectory is used. It is also divided into four segments according to the same logic. Subsequently, the trajectory segment to be evaluated and the standard trajectory segment of each action stage are time-sequentially aligned. The preferred method is to use key node anchoring combined with dynamic time warping to complete the intra-stage alignment.

[0148] Step 7: Stage Scoring. Taking the transportation stage as an example, the trajectory of this stage is resampled into L=10 three-dimensional sampling points by arc length normalization, and the spatial shape consistency index is calculated. Turning points and intermediate transition points are used as key nodes to calculate the hit index of key action nodes. ; Calculate the three-dimensional motion trend consistency index using the displacement vectors of the starting and ending points. The spatial smoothing bias penalty index is calculated using the three-dimensional second-order difference average. If we assume , , , The score for the transportation phase is: .

[0149] Step 8: Overall Score. Calculate the scores for each of the four action phases. , , , Then by weight , , , The weighted summaries are then used to obtain the comprehensive three-dimensional trajectory score. .

[0150] In this embodiment, if the spatial deviation of key nodes in the grasping stage is large, then Lower, leading to Decrease; if there is significant shaking during the handling stage, then Enlargement, leading to The error rate decreases. Teachers can use this information to determine whether the main problem in the task lies in the grasping accuracy or the stability of the handling, and then provide targeted guidance.

[0151] This complete embodiment demonstrates that the present invention does not simply convert a two-dimensional trajectory into a three-dimensional one, but rather obtains a scoreable three-dimensional trajectory through multi-constraint solving in the constrained scenario of a teaching robot, and further forms a staged and interpretable trajectory scoring result.

[0152] This invention also provides a three-dimensional evaluation system for the operation trajectory of a teaching robot, including a data acquisition module, a three-dimensional mapping module, a trajectory scoring module, and an output module, such as... Figure 5 As shown.

[0153] The data acquisition module is used to acquire image sequences of the teaching robot performing practical tasks captured by a fixed external camera, and to identify and continuously track the end effector, tool center point or preset key parts of the teaching robot to obtain two-dimensional operation trajectory information.

[0154] The 3D mapping module is used to map each 2D trajectory point in the 2D operation trajectory information into a spatial ray based on the camera's intrinsic and extrinsic parameters. It also uses the spatial depth parameter corresponding to each trajectory point as an optimization variable to construct a constrained optimization model for mapping 2D trajectory points to 3D trajectory points in the world coordinate system. Based on the constrained optimization model, it solves for the spatial position of each trajectory point in the world coordinate system to generate 3D operation trajectory information.

[0155] The trajectory scoring module is used to divide the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information into action stages and align them in time according to the key events, key action nodes or preset spatial regions corresponding to the practical task. For each action stage, a stage matching score is generated based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency and spatial smoothness deviation. The trajectory scoring result is generated based on the matching score of each action stage.

[0156] The output module is used to display, store, or transmit trajectory scoring results.

[0157] The 3D mapping module may further include a camera calibration subunit, a ray construction subunit, a constraint modeling subunit, and an optimization solution subunit; the trajectory scoring module may further include a stage division subunit, a temporal alignment subunit, a stage index calculation subunit, and a comprehensive score calculation subunit.

[0158] As can be seen from the above description, the above embodiments of the present invention map the key steps in the method into clear and deployable functional units through a modular structure, which facilitates integration with teaching robot platforms, vision terminals and evaluation interfaces.

[0159] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, it implements the method described in any of the above embodiments.

[0160] The electronic device may be a teaching robot control terminal, an edge computing box, an industrial control computer, a server, a laptop computer, or other devices with data processing capabilities. The memory may include read-only memory, random access memory, flash memory, solid-state drive, etc.

[0161] In one embodiment, the data acquisition module runs on an industrial computer and is connected to a fixed external industrial camera via a gigabit Ethernet port or USB 3.0 interface for data acquisition. Preferably, a fixed external monocular industrial camera is used to acquire image sequences; without changing the core technical approach of the present invention, a binocular camera or depth camera can also be used as optional visual acquisition devices, but the two-dimensional trajectory points in the image sequence are still used as the basic observation information, and additional depth information or multi-view observation information can be introduced into the optimization solution process as auxiliary constraints. The industrial computer can also synchronously receive key task triggering events sent by the underlying PLC or robot control cabinet, such as gripper closure, stage switching, action completion instructions, etc., thereby achieving spatiotemporal alignment of visual trajectory data and control logic events.

[0162] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0163] In one optional implementation, the world coordinate system can be selected as the camera coordinate system, the robot base coordinate system, or a coordinate system unified by the two through extrinsic parameter transformation; in another optional implementation, if the teaching platform synchronously provides joint encoder data, the encoder data can be incorporated into the overall optimization objective as additional constraint information to further improve the accuracy of the three-dimensional trajectory solution.

[0164] In one alternative implementation, in addition to numerical values, the scoring results may also output a stage radar chart, a list of key node deviations, a spatial trajectory visualization, and suggestive text descriptions to help teachers or users understand the reasons for the scoring.

[0165] This invention revolves around a core technology chain of two-dimensional operation trajectory, three-dimensional operation trajectory in the world coordinate system, and staged matching and scoring of preset standard three-dimensional trajectory, forming a logically complete and directly implementable technical solution.

Claims

1. A three-dimensional evaluation method for the operating trajectory of a teaching robot, characterized in that, include: The system acquires image sequences of the teaching robot performing practical tasks from a fixed external camera, identifies and continuously tracks the end effector, tool center point, or preset key parts of the teaching robot, and obtains two-dimensional operation trajectory information. Based on the camera's intrinsic and extrinsic parameters, each two-dimensional trajectory point in the two-dimensional operation trajectory information is mapped to a spatial ray. The spatial depth parameter corresponding to each trajectory point is used as an optimization variable to construct a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system. Based on the constrained optimization model, the spatial position of each trajectory point in the world coordinate system is obtained to generate three-dimensional operation trajectory information. The camera's intrinsic and extrinsic parameters include at least the camera intrinsic parameter matrix, rotation matrix, and translation vector; the constrained optimization model includes at least three of the following: projection consistency constraint, robotic arm reachable space constraint, task area constraint, key action position constraint, and trajectory smoothing constraint. Based on the key events, key action nodes, or preset spatial regions corresponding to the practical task, the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information are divided into action stages and aligned in time sequence. The criteria for dividing the action phase include at least one of the following: task triggering events output by the robot controller or PLC, changes in the state of the end effector, the trajectory entering a preset space area, changes in the speed threshold, and the hit time of key action nodes. For each action stage, a stage matching score is generated based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency, and spatial smoothness deviation. Trajectory scoring results are then generated based on the matching scores for each action stage. Output the trajectory score results.

2. The three-dimensional evaluation method for the operation trajectory of a teaching robot according to claim 1, characterized in that, The process involves mapping each two-dimensional trajectory point in the two-dimensional operation trajectory information to a spatial ray based on the camera's intrinsic and extrinsic parameters, and constructing a constrained optimization model for mapping two-dimensional trajectory points to three-dimensional trajectory points in the world coordinate system using the spatial depth parameter corresponding to each trajectory point as an optimization variable. Based on this constrained optimization model, the spatial position of each trajectory point in the world coordinate system is obtained to generate three-dimensional operation trajectory information, including: Determine the spatial ray direction vector in the camera's normalized plane based on the two-dimensional trajectory points and the camera's intrinsic parameter matrix; Establish the correspondence between 3D points in the camera coordinate system and the spatial ray direction vector and spatial depth parameters; Transform 3D points in the camera coordinate system to the world coordinate system based on the camera's extrinsic parameters; A constrained optimization model is constructed to describe the mapping relationship between two-dimensional trajectory points and three-dimensional trajectory points in the world coordinate system. Based on the constrained optimization model, the spatial depth parameters corresponding to each trajectory point are obtained to generate the three-dimensional operation trajectory information.

3. The three-dimensional evaluation method for the operation trajectory of a teaching robot according to claim 1, characterized in that, Based on the key events, key action nodes, or preset spatial regions corresponding to the practical task, the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information are divided into action stages and aligned temporally. A trajectory scoring result is generated based on the matching score of each action stage, including: Based on the key events, key action nodes, or preset spatial regions corresponding to the practical task, the three-dimensional operation trajectory obtained through the constraint optimization model and the preset standard three-dimensional trajectory are divided into action stages. The timing of each action phase is aligned, and a trajectory matching score for each phase is generated based on the matching results of each action phase. The trajectory scoring result is generated based on the trajectory matching score at each stage.

4. The three-dimensional evaluation method for the operation trajectory of a teaching robot according to claim 3, characterized in that, The matching status of each action stage includes at least two or more of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency, and spatial smoothness deviation.

5. The three-dimensional evaluation method for the operation trajectory of a teaching robot according to claim 4, characterized in that, The first of each action phase The matching score for each action phase is determined as follows: in, Indicates the first Matching score for each action phase Indicators representing spatial shape consistency This indicates that key action nodes have been hit by the indicator. Indicators representing the consistency of three-dimensional motion trends This represents the spatial smoothing deviation penalty index. , , , This represents the weighting coefficient of the corresponding indicator.

6. The three-dimensional evaluation method for the operation trajectory of a teaching robot according to claim 3, characterized in that, The trajectory scoring result includes a comprehensive three-dimensional trajectory score, which is determined as follows: in, This represents the comprehensive three-dimensional trajectory score. Indicates the number of action stages. Indicates the first The scoring weights corresponding to each action phase Indicates the first Matching score for each action phase.

7. A three-dimensional evaluation system for the operating trajectory of a teaching robot, characterized in that, include: The data acquisition module is used to acquire image sequences of the teaching robot performing practical tasks captured by a fixed external camera, and to identify and continuously track the end effector, tool center point or preset key parts of the teaching robot to obtain two-dimensional operation trajectory information. The 3D mapping module is used to map each 2D trajectory point in the 2D operation trajectory information into a spatial ray based on the camera's intrinsic and extrinsic parameters. It also uses the spatial depth parameter corresponding to each trajectory point as an optimization variable to construct a constrained optimization model for mapping 2D trajectory points to 3D trajectory points in the world coordinate system. Based on the constrained optimization model, it solves for the spatial position of each trajectory point in the world coordinate system to generate 3D operation trajectory information. The camera's intrinsic and extrinsic parameters include at least the camera intrinsic parameter matrix, rotation matrix, and translation vector; the constrained optimization model includes at least three of the following: projection consistency constraint, robotic arm reachable space constraint, task area constraint, key action position constraint, and trajectory smoothing constraint. The trajectory scoring module is used to divide the three-dimensional operation trajectory information and the preset standard three-dimensional trajectory information into action stages and align them in time according to the key events, key action nodes or preset spatial regions corresponding to the practical task. For each action stage, a stage matching score is generated based on at least two of the following: spatial shape consistency, key action node hit status, three-dimensional motion trend consistency and spatial smoothness deviation. The trajectory scoring result is generated based on the matching score of each action stage. The criteria for dividing the action phase include at least one of the following: task triggering events output by the robot controller or PLC, changes in the state of the end effector, the trajectory entering a preset space area, changes in the speed threshold, and the hit time of key action nodes. The output module is used to output the trajectory scoring results.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, it implements the three-dimensional evaluation method for the operating trajectory of the teaching robot as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional evaluation method for the operating trajectory of the teaching robot as described in any one of claims 1 to 6.

Images