Mechanical arm motion trail detection method and system and program product

Abstract

The invention provides a mechanical arm motion trail detection method and system and a program product, and relates to the technical field of production management. The method comprises the following steps: acquiring a to-be-processed image with a target mechanical arm sent by a binocular camera device; detecting the to-be-processed image by adopting a pre-established key point detection model to obtain at least one attitude key point and a two-dimensional coordinate corresponding to the attitude key point; performing trajectory smoothing on the two-dimensional coordinates by adopting a preset filtering model to obtain a key point trajectory; performing parallax processing and triangulation on attitude key points on the key point track to obtain three-dimensional coordinates of the attitude key points and a target motion track containing the three-dimensional coordinates; and the target movement track is compared with a preset movement track, and a track analysis result used for indicating whether the target mechanical arm is abnormal or not is obtained according to a comparison result. The accuracy and timeliness of monitoring the motion trail of the mechanical arm are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of production management, and in particular to a method, system and program product for detecting a motion trajectory of a robot arm. Background Art

[0002] As the most widely used automation equipment in robotics, robotic arms have demonstrated their outstanding capabilities in many fields, including industrial production, medical surgery, entertainment services, military operations, semiconductor manufacturing, and space exploration. Although robotic arms have different appearances and structures, they all share a significant feature: they can receive instructions, accurately locate a point in three-dimensional (or two-dimensional) space, and perform corresponding tasks. This high degree of accuracy and flexibility makes robotic arms an indispensable part of modern automation, bringing great convenience and efficiency to all aspects of our lives.

[0003] However, in the actual production environment, the motion trajectory of the robot may be abnormal, which is mainly affected by factors such as the robot's hardware, software, control system, and external environment. At present, in order to detect robot abnormalities in the production environment, the following methods are usually used: First, through forward and inverse kinematics, mathematical modeling is implemented to calculate the posture of each joint of the robot, which has good flexibility and real-time performance, but when the robot program fails, the motion trajectory of the robot cannot be detected, resulting in serious production accidents. Second, the robot trajectory is detected by machine vision algorithm, but it is easily disturbed by factors such as light, occlusion, and environment, which affects the accuracy of the algorithm. Therefore, the accuracy and stability of the existing robot motion trajectory monitoring method are low, resulting in poor monitoring effect. Summary of the invention

[0004] The present invention provides a robot arm motion trajectory detection method, system and program product, which are used to solve the technical problem that the existing robot arm motion trajectory monitoring method has low accuracy and stability, resulting in poor monitoring effect.

[0005] In a first aspect, the present application provides a method for detecting a motion trajectory of a robot arm, for use in a processing device, the method comprising:

[0006] Acquire an image to be processed having a target robotic arm sent by a binocular camera device;

[0007] Using a pre-established key point detection model to detect the image to be processed, and obtaining at least one posture key point and its corresponding two-dimensional coordinates;

[0008] Using a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory;

[0009] Performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates;

[0010] The target motion trajectory is compared with a preset motion trajectory, and a trajectory analysis result indicating whether the target robotic arm is abnormal is obtained according to the comparison result.

[0011] In a second aspect, the present application provides a robot arm motion trajectory detection system, including: a binocular camera device and a processing device:

[0012] The binocular camera device is used to obtain the image to be processed with the target robotic arm, and then send it to the processing device;

[0013] The processing device is connected to the binocular camera device and is used to obtain the image to be processed;

[0014] Using a pre-established key point detection model to detect the image to be processed, and obtaining at least one posture key point and its corresponding two-dimensional coordinates;

[0015] Using a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory;

[0016] Performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates;

[0017] The target motion trajectory is compared with a preset motion trajectory, and a trajectory analysis result indicating whether the target robotic arm is abnormal is obtained according to the comparison result.

[0018] In a third aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the robot arm motion trajectory detection method described in the first aspect or any possible implementation method of the first aspect.

[0019] The robot arm motion trajectory detection method, system and program product provided by the present application detect the posture key points of the target robot arm by combining a binocular camera device and a key point detection model, thereby ensuring high precision and high stability of key point detection, and can be applied to complex industrial environment scenes and situations where multiple robot arms work simultaneously. Then, the two-dimensional coordinates of the posture key points are smoothed to reduce detection interference. Furthermore, through binocular vision processing and triangulation of the binocular camera device, the two-dimensional coordinates of the posture key points are converted into three-dimensional coordinates, which can restore spatial dimension information and form a corresponding target motion trajectory, thereby improving the accuracy of robot arm trajectory detection. Thus, the target motion trajectory marked with the three-dimensional coordinates of the posture key points is compared with the preset motion trajectory, so as to realize timely detection of robot arm abnormalities in the production environment, and solve the problem that the accuracy and stability of the existing robot arm motion trajectory monitoring method are low, resulting in poor monitoring effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of a method for detecting a motion trajectory of a robotic arm provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the structure of a YOLOX-Pose model provided for an exemplary embodiment of the present application;

[0022] Figure 3 A schematic diagram of a flow chart of a method for calculating three-dimensional coordinates of a target key point provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the principle of parallax processing provided for an exemplary embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a robotic arm motion trajectory detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Figure 1 A schematic flow chart of a method for detecting a motion trajectory of a robotic arm provided in an embodiment of the present application, wherein the method is used in a processing device and comprises steps S101 to S105.

[0027] S101, obtaining an image to be processed having a target robotic arm sent by a binocular camera device.

[0028] It should be noted that the binocular camera device can be a binocular camera. The binocular camera device of the present application must meet the industrial production needs and algorithm imaging requirements. The binocular camera device needs to have sufficient resolution to clearly capture the details of the robot arm. The resolution of the binocular camera device needs to be related to the object imaging size and the detection accuracy. The frame rate of the binocular camera device should be high enough to capture the rapid movement of the robot arm, especially when real-time monitoring and feedback control are required. If the robot arm moves faster, it needs to be selected. The binocular camera device should be able to provide good contrast and color reproduction for subsequent image processing and analysis. The field of view of the lens of the binocular camera device should be large enough to cover the working space of the robot arm, but at the same time, sufficient resolution must be ensured. If multiple binocular camera devices are used for detection, they need to be able to work synchronously to ensure that the motion data captured at different viewing angles is consistent. The system of the binocular camera device needs to be sufficiently stable to reduce image blur caused by vibration or movement. The binocular camera device should be able to work under different lighting conditions, including environments with dim light or large light changes.

[0029] In addition, the binocular camera device needs to have a good interface (such as USB, GigE, Camera Link, etc.) to facilitate connection with the computer system, and needs to have corresponding software support, including drivers and image processing libraries. In industrial environments, binocular camera devices may require additional protection measures, such as dustproof, waterproof, vibration-resistant, etc., to ensure their reliability in harsh environments.

[0030] It should be noted that the binocular camera device uses a calibration plate with a known pattern, usually a black and white checkerboard. The size of the checkerboard needs to be known because it will be used for subsequent size calibration. Place the calibration plate in the field of view of the two cameras, and take multiple images from different angles and distances, with each camera taking more than three images. Make sure that the pattern of the calibration plate occupies a large enough area in the image, and that there are multiple different perspectives in the image, so that the checkerboards of multiple images can cover every pixel in the field of view of the binocular camera device. Detect the corner points of the checkerboard in the captured image. These corner points will be used for subsequent calibration calculations.

[0031] Each camera device is calibrated separately to calculate its internal parameters (focal length, principal point, distortion coefficient, etc.) and external parameters (position and posture of the camera device relative to the calibration plate). This step usually uses Zhang's calibration method or checkerboard calibration method, and then the obtained internal and external parameters are subjected to distortion correction. Based on the calibration of a single camera device, binocular correction is performed so that the same object is the same size in the two images and is horizontally in a straight line. After determining the correspondence between the three-dimensional space and the image through calibration, the disparity is calculated through the correspondence between the points in the three-dimensional space on the left and right images. This process is the purpose of stereo matching. Through stereo matching technology, the correspondence between the points in the left and right images can be clarified, thereby obtaining the disparity and restoring the three-dimensional information of the points.

[0032] In the present application, the robot arm is a mechanical device that can simulate the movement of a human arm and has the ability to extend, rotate and grasp objects. That is to say, the present application obtains an image of the target robot arm through a binocular camera device, and then performs key point detection on the image through a key point detection model to obtain at least one posture key point, so that the motion trajectory of the target robot arm is formed by the posture key point, so as to realize timely detection of robot arm abnormalities in a production environment and improve the real-time monitoring.

[0033] S102: Use a pre-established key point detection model to detect the image to be processed to obtain at least one posture key point and its corresponding two-dimensional coordinates.

[0034] In the present application, the key point detection model can be a neural network model with the function of detecting target feature points / key points, such as a YOLOV5 detector, or an improved YOLO-Pose model based on the YOLOV5 detector. Specifically, the key points of the robotic arm are pre-set, and the key points can be components in the mechanical structure for aspects such as degrees of freedom, load capacity, accuracy, and speed. For example, one or more of the end of the robotic arm, the arm rod of the robotic arm, and the joint point of the robotic arm are used as key points. Then, the key point detection model is trained using data about the key points to input the currently collected robotic arm image into the trained key point detection model, and output the key points on the robotic arm image. Therefore, the present application combines a binocular camera device and a key point detection model to detect the key points of the posture of the target robotic arm, thereby ensuring high accuracy and high stability of key point detection.

[0035] For example, the YOLO-Pose model introduces the YOLOV5 algorithm structure into key point detection. The algorithm has advanced performance, good real-time performance, and end-to-end detection is easy to deploy. In industrial environments, in order to meet higher detection accuracy, the camera resolution is often large, which adds a lot of calculations. The key point detection of the robotic arm has the characteristics of simple scenes, fewer targets, and a relatively fixed field of view and fewer key points. The parameters of the model can be reduced to ensure that the calculation amount of the model is greatly reduced without losing accuracy, which is more suitable for the detection of industrial robotic arms. In addition, in the process of binocular vision parallax, the target box can be used to extract the area of ​​interest from the image for feature point detection and matching to further reduce the amount of calculation.

[0036] S103: Use a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory.

[0037] In the present application, in the acquired robotic arm video, the movement change of the key points of the robotic arm's posture can be regarded as a linear movement in a two-dimensional image, which meets the filter usage standard, thereby forming a key point trajectory. Among them, the preset filtering model can be a model with a filtering algorithm, such as a Kalman filtering model.

[0038] It should be noted that when the Kalman filter algorithm (i.e., the Kalman filter model) is used to predict the key points of the robot's posture, the focus is on implementing comprehensive and effective feedback control of the state observation system, which specifically includes two important processes: prediction and update. Among them, the key to prediction is to sort out and analyze the observed values ​​of the system state and error, so as to make a scientific and reasonable prior estimate of the state at the next moment; updating means that after obtaining the observed value at this moment, the observed value is compared with the prior estimate, so as to complete the real-time update of the Kalman filter. By repeating the above two processes, the reasonable prediction of the key points of the robot can be completed through the Kalman filter.

[0039] Specifically, the Kalman filter describes the system through two parts: the state equation and the observation equation. The system state can be described by the following formula; X K =AX K-1 +BU K +W K-1 Among them, X K is the state variable at time K, X K-1 is the state variable at time K-1, A is the state transfer matrix of the system, B is the input conversion matrix, W K , W K-1 is white noise (i.e., the process noise of the system), U K is the control input at time K.

[0040] The system observation equation is: K =H K XK +V K Among them, Z K is the system observation value at time K, H K is the observation matrix at time K, which is usually established as a constant, V K is a white noise sequence with a mean of 0 (i.e., the measurement noise at time K), R is the covariance matrix of the observation equation, and W K That is, the process noise W K-1 It obeys normal distribution, and its probability density function is P(w)~N(0,Q), with a mean of 0, and Q is the covariance matrix of process noise. Measurement noise V K It also obeys the normal distribution, and its probability density function is P(v)~N(0,R), with a mean of 0, and R is the covariance matrix of the observation equation.

[0041] Known current state X K , the prior state estimate before time K and the posterior state estimate at time K The error formulas can be obtained as follows: in, is the prior estimation error, e K is the posterior estimation error. The covariance of the prior estimation error is derived and the covariance P of the posterior estimation error K They are as follows: in, is the matrix transpose of the prior estimation error, is the matrix transpose of the posterior estimation error. By calculation and derivation, we can get the posterior estimation As shown below: Where K K is the gain matrix, H is the observation matrix, H T is the matrix transpose of the observation matrix, which can be calculated by the following formula:

[0042] The Kalman filter formula can be quickly and easily derived from the above formula: state prediction equation: Among them, μ K is the control input at time K. Covariance prediction equation: Among them, AT is the matrix transpose of the system's state transfer matrix, P K-1 is the covariance of the posterior estimation error at time K-1. Kalman gain equation: State update equation: Covariance update equation: Where I is the identity matrix.

[0043] Therefore, the present application obtains the two-dimensional coordinates of the key points of the posture, smoothes the trajectory through the Kalman filter algorithm, obtains the key point trajectory, reduces the trajectory jitter caused by detection, and reduces detection interference.

[0044] S104, performing parallax processing and triangulation measurement on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates.

[0045] It should be noted that parallax refers to the position difference between corresponding pixel points of the same object on the imaging plane of the left and right cameras (or eyes), that is, the parallax in this application can be the position difference between the left image and the right image of any posture key point in the image to be processed obtained by the binocular camera device. Then, using the principle of triangulation, combined with the parallax and the intrinsic matrix and external parameters of the camera, the three-dimensional coordinates of the posture key point are calculated, and the target motion trajectory of the posture key point with three-dimensional coordinates is formed based on the key point trajectory.

[0046] Therefore, the present application uses binocular vision and triangulation to restore the two-dimensional coordinates of the posture key points to three-dimensional coordinates, so as to compare the three-dimensional coordinates with the preset motion trajectory, and can more accurately monitor the motion trajectory of the robotic arm.

[0047] S105 , comparing the target motion trajectory with a preset motion trajectory, and obtaining a trajectory analysis result indicating whether the target robotic arm is abnormal according to the comparison result.

[0048] In the present application, the posture of the robotic arm is solved according to the three-dimensional coordinates of the posture key points, and by comparing with the motion trajectory of each joint of the preset normal robotic arm, it is possible to determine whether there is an abnormality in the robotic arm. Optionally, in order to ensure that there is a higher detection accuracy between the preset motion trajectory and the target motion trajectory of the robotic arm, the preset motion trajectory can also obtain the standard data of each key point through the same algorithm. Then, the motion position state of each key point of the robotic arm at each moment is first compared with the preset motion trajectory, and a detection threshold is set. If the trajectory deviates from the preset motion trajectory by more than this threshold, it is determined that a trajectory abnormality occurs at a certain motion key point of the robotic arm. Exemplarily, by comparing the state between the target motion trajectory and the preset motion trajectory, it is determined whether the robotic arm is abnormal. Specifically, the state may include a displacement state, a velocity state, and an acceleration state. For example, the velocity state is expressed as the following formula: Among them, v t+1 is the velocity state at time t, x t+1 is the position at time t+1, x t is the position at time t, and δt is the time interval between each frame. The acceleration state is expressed as the following formula: Among them, a t+2 is the acceleration state at time t+2, x t+2 is the position at time t+2.

[0049] Optionally, the state analysis data used to compare the trajectory can be obtained after weighted calculation of at least one of the displacement state, velocity state and acceleration state, and the weight corresponding to each state can be determined based on the task requirements, environmental conditions, etc. of the target task to be performed by the robot arm. The present application does not make any limitation on this. The number of parameters introduced in each state gradually increases, and the number of judgment dimensions increases, making the judgment more stable.

[0050] Therefore, this embodiment can obtain the state information such as displacement, velocity and acceleration of the robot arm through three-dimensional coordinates, and then compare the state information with the normal trajectory for real-time detection, so as to generate an alarm in time when the motion state of the robot arm deviates, while ensuring the accuracy of the detection.

[0051] In some embodiments, the construction of the key point detection model includes the following steps:

[0052] Acquire a training data set, mark a bounding box of image training data in the training data set and key point positions of corresponding key points, wherein the key point includes at least one joint point on the target robotic arm;

[0053] Get the preset target detection model;

[0054] Constructing a target detection loss function based on the labeled training data set, wherein the target detection loss function is used to indicate training the bounding box;

[0055] Based on the labeled training data set and the preset OKS indicator, a first key point loss function and a second key point loss function are constructed, wherein the first key point loss function is used to indicate the evaluation of the key point by the training, and the second key point loss function is used to indicate the confidence of the key point in the training;

[0056] Based on the labeled training data set and the target detection loss function, construct a sorting and classification loss function, wherein the sorting and classification loss function is used to indicate the sorting and classification of samples in the training bounding box;

[0057] Determine a model loss function based on the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function;

[0058] The target detection model is trained based on the model loss function to obtain a key point detection model.

[0059] In this embodiment, the key point detection model can be a YOLO-Pose model. The YOLO-Pose model is developed based on the YOLOV5 detector, but this causes YOLO-Pose to have the same problem as YOLOV5: the problem of inconsistency between classification and regression in training and prediction of a single-stage target detector, that is, YOLOV5 is trained separately during classification and regression training, and the loss is calculated in parallel and back-propagation optimization is performed. However, in the prediction stage, the samples are preliminarily screened by classification, and then the final screening is serial processing through non-maximum suppression. This may result in a high classification score, but the frames with poor regression will be retained. When detecting the key points of the robotic arm, the sample screening will not be thorough enough, and unnecessary targets and key points will appear. Therefore, this embodiment combines the Rank&Sort (RS Loss) loss function (i.e., the sorting classification loss function) to eliminate the inconsistency problem between classification and regression, and further enhance the algorithm recognition ability.

[0060] The YOLOX-Pose model adds key point prediction to each stage of the YOLOV5 model, detects each instance and predicts the key points of each instance on this basis. With almost no increase in the amount of computation, it solves the problems of two-stage heat maps that are not end-to-end training and difficult to deploy, training relies on the alternative L1 loss, there is no maximization evaluation strategy, and it is difficult to handle key points that are close in distance and difficult to predict.

[0061] For example, Figure 2 A schematic diagram of the structure of a YOLOX-Pose model provided by an exemplary embodiment of the present application. The YOLOX-Pose model includes a backbone network arranged in sequence ( Figure 2 Backbone), neck network ( Figure 2 Neck) and multiple head networks ( Figure 2 Each head network corresponds to the target box prediction network ( Figure 2 Box) and key point prediction network ( Figure 2 Specifically, the image to be processed with the target robotic arm is input into the YOLOX-Pose model, and the feature extraction of the image to be processed is performed through Backbone to obtain feature maps of different scales. Then, the feature maps of different scales are input into Neck to perform feature fusion and obtain fused features. Furthermore, the fused features are input into each head network to predict the robotic arm bounding box and key points.

[0062] In this embodiment, each robot has multiple joints, which depends on the number of movable joints of the robot. Each joint is set as a key point, and the images in the training data set are marked. The position of each key point (i.e., the output is a posture key point) is represented as x, y, and the confidence of each key point is represented as conf. The number of key point prediction channels for each instance is n×3. Therefore, the output channel of the YOLO-Pose model is:

[0063]

[0064] Among them, P v is the prediction vector output by the YOLO-Pose model, C x is the horizontal coordinate of the center point of the robot arm bounding box, C y is the ordinate of the center point of the robot arm bounding box, W is the width of the robot arm bounding box, H is the height of the robot arm bounding box, box conf is the confidence of the robot's bounding box, class conf is the classification confidence, n is the number of key points, represents the horizontal coordinate of the i-th key point, represents the ordinate of the i-th key point, Represents the confidence of the i-th key point.

[0065] In this embodiment, most current target detectors use IoU variants, such as GIoU, DIoU, and CIoU losses, instead of distance-based losses, to better detect bounding boxes. These losses are scale-invariant and can directly optimize the evaluation strategy itself. YOLO-Pose uses the CIoU loss function for bounding box supervision. In order to combine the Rank&Sort loss function (i.e., the sorting classification loss function) and standard consistency, the GIoU loss function (i.e., the target detection loss function) is used. Therefore, the target detection loss function is expressed as follows:

[0066]

[0067] Where s is the scale of the bounding box, i is the horizontal coordinate of the position of the bounding box, and j is the vertical coordinate of the position of the bounding box; L box (s, i, j, k) represents the loss of the bounding box matched to the kth instance at position (i, j) and scale s; GIoU represents the GIoU loss function; represents the GT bounding box of the kth instance at position (i, j) and scale s, represents the predicted bounding box of the k-th instance at position (i, j) and scale s.

[0068] Furthermore, in addition to optimizing the IoU evaluation strategy itself, the YOLO-Pose model of this embodiment also optimizes the key point evaluation strategy OKS. The L1 loss used by the bottom-up heat map method to detect key points is not suitable for obtaining the optimized OKS indicator. Because the L1 loss does not consider the scale of an object or the type of key points, and the heat map is a probability map, it is impossible to use the OKS indicator as the loss function of the heat map method. In the YOLO-Pose model, using the OKS indicator as the IoU loss is scale-invariant. The OKS loss (i.e., the first key point loss function) calculates the OKS indicator for each individual key point position and accumulates it to the final OKS loss or key point IoU loss, i.e.:

[0069]

[0070] Among them, L kpts_pos (s, i, j, k) is denoted as the first keypoint loss for the keypoint matched to the kth instance at position (i, j) and scale s; N kpts represents the number of key points, d n Indicates the distance between the predicted position and the true position of the nth key point; k n represents the key point weight; s represents the scale; v n Indicates whether the nth key point is visible, δ(v n ) indicates the display mark of each key point, v n >0 means that the nth key point is visible.

[0071] At the same time, for each key point, a confidence parameter is set to predict whether the instance has the key point. Therefore, the mark of whether the key point is visible is used as the real mark to construct the second key point loss function. Specifically, the second key point loss function is expressed as:

[0072]

[0073] Among them, L kpts_cof (s, i, j, k) is denoted as the second keypoint loss for the keypoint matched to the kth instance at position (i, j) and scale s, Represents the prediction confidence of the nth key point.

[0074] Furthermore, in this embodiment, although the GIoU evaluation strategy is used to calculate the target detection loss function, it is impossible to avoid the situation where the IoU score of the high classification instance is low and the IoU score of the low classification score is high, which makes it easy to have no recognition when screening samples in post-processing, resulting in large errors in key point detection. Therefore, the Rank&Sort loss is introduced to sort the IoU, so as to screen instances with higher classification scores and higher IoU scores in post-processing. RS Loss does not add additional auxiliary heads to solve the problem of inconsistency between training and prediction, and only uses RS Loss to perform back propagation supervision on the model. It uses IoU as a classification label so that positive samples in the predicted bounding box can be sorted by continuous values ​​(IoU scores) as labels. In this way, all positive samples can be sorted before negative samples, and only negative samples with higher scores are selected for calculation, which solves the problem of imbalance between positive and negative samples without using heuristic sampling, and does not require multi-task weight or coefficient adjustment. Therefore, the ranking classification loss function is constructed as follows: Among them, P represents a set of positive examples, l RS (i) represents the cumulative sum of the current ranking error (rank error) and classification error (sort error) (i.e., the current ranking error), represents the target sorting error, L RS It is the average value of the difference between the current sorting error and the target sorting error on the positive examples (sorting classification loss).

[0075] Based on the above embodiments, in some embodiments, determining the model loss function based on the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function includes:

[0076] The model loss function is determined by the following formula (1):

[0077] L=∑ s，i，j，k (L RS +λ cls L cls +λ RS_box L box +λ RS_box L kpts_pos +λ RS_box L kpts_conf ) (1);

[0078] Wherein, L is the model loss function, s represents the preset scale factor, i represents the first position parameter in the position (i, j), j represents the second position parameter in the position (i, j), k represents the instance, and L RS is the ranking classification loss function, Lcls is the pre-acquired classification loss function, L box is the target detection loss function, L kpts_pos is the first key point loss function, L kpts_conf is the second key point loss function, λ cls is the first loss function hyperparameter about the classification loss function, λ RSb0x is the second loss function hyperparameter;

[0079] The second loss function hyperparameter is determined by the following formula (2):

[0080] λ Rs_box =|L Rs / L box | (2).

[0081] Therefore, this embodiment performs weighted calculation on the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function to determine the model loss function, thereby performing model training. Therefore, this embodiment trains the key point detection model through the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function, thereby improving the detection accuracy of the bounding box and key points of the robot arm, achieving the characteristics of using specific scenarios at a higher resolution to greatly reduce the amount of model calculation, and solving the problem of erroneous detection caused by inconsistency between training and post-processing. This embodiment has high real-time performance and accuracy, and reduces the loss weight hyperparameters, and can detect the key points of the robot arm in real time in an industrial environment.

[0082] In some embodiments, Figure 3 The flowchart of the method for calculating the three-dimensional coordinates of the target key points provided in the embodiment of the present application is schematic. The method for calculating the three-dimensional coordinates of the target key points indicated in step S104 includes steps S301 to S303.

[0083] S301, based on the operation task and / or environmental data of the target manipulator, screening the target key points corresponding to the posture key points on the key point trajectory when they meet the preset screening conditions, wherein the operation task includes the operation type and the state characteristic data of the operation object;

[0084] S301, calculating the disparity of the target key point in the image to be processed;

[0085] S301 . Triangulate the target key point based on the parallax to obtain the three-dimensional coordinates of the target key point.

[0086] In this embodiment, the two-dimensional coordinates of the posture key points are smoothed by a preset filtering model to obtain the key point trajectory. At this time, the target key points are screened out according to the operation task and the environment, and the key points with higher quality are further extracted to ensure that the key point data used in the subsequent calculation and analysis have higher quality and accuracy. For example, the screening is performed by the response strength, stability or repeatability of the feature points at different scales. Then, the target key points are processed by parallax and triangulated to obtain the three-dimensional coordinates of the target key points, which can reduce the number of target feature points calculated, thereby significantly reducing the calculation complexity, improving the key point detection speed, and realizing timely detection of the movement of the robot arm.

[0087] It should be noted that the operation tasks include but are not limited to material handling (such as material handling on the production line, including grabbing, lifting, moving and placing parts, products, boxes, etc.), processing and manufacturing (such as assembly, welding, cutting, grinding, etc.), quality inspection (such as dimension measurement, defect detection, etc.), household services (such as cleaning, cooking), etc., that is, the corresponding operation types (such as handling, grabbing, processing and manufacturing, etc.) and operation objects are extracted based on the operation tasks. The state feature data of the operation object includes dimensional data (such as external dimensions, weight, etc.), motion state (such as motion, stillness), posture data, target position data, motion parameters and the corresponding motion trajectory of the operation object.

[0088] For example, Figure 4 A schematic diagram of the principle of parallax processing provided by an exemplary embodiment of the present application, where the distance between the projection centers of the two cameras is known to be b, the focal length of the camera is f, and the imaging point of any point P in the three-dimensional space on the left camera is P L , at the imaging point PR of the right camera. According to the principle of rectilinear propagation of light, the three-dimensional space point P is the intersection of the line connecting the projection center points of the two cameras and the imaging point. Figure 4 As shown, line segment X L and X R are the distances from the imaging point of the left and right cameras to the left imaging plane, respectively. The parallax d of point P in the left and right cameras (left image and right image) can be defined as follows: d = |X L -X R |. Two imaging points P L , P R The distance between them is: P L P R =b-(X L -X R ). According to the theory of similar triangles, we can get the distance Z from point P to the center plane of the projection: It is known that the image plane coordinates of point P in three-dimensional space, which are offset (x0, y0) from the image coordinate origin relative to the imaging center, are (x, y). The three-dimensional coordinates of the point are obtained as follows: Among them, X is the horizontal coordinate of the three-dimensional coordinate of point P, and Y is the vertical coordinate of the three-dimensional coordinate of point P.

[0089] Therefore, this embodiment screens out high-quality target key points through operating tasks and the environment, performs parallax processing and triangulation on the target key points, and obtains the three-dimensional coordinates of the target key points, thereby reducing the number of calculated target feature points and ensuring that the key point data used in subsequent calculations and analysis has high quality and accuracy. At the same time, it can significantly reduce the calculation complexity, improve the key point detection speed, and realize timely detection of the movement of the robotic arm.

[0090] Based on the above embodiments, in one embodiment, according to the operation task of the target robot arm, the target key points corresponding to the posture key points that meet the preset screening conditions are screened, including:

[0091] Based on the operation type of the target robotic arm, determining the operation correlation between the posture key point and the operation type;

[0092] Acquire an operation image having the operation object sent by the binocular camera device, so as to identify state feature data of the operation object based on the operation image, wherein the state feature data includes posture data, target position data, motion parameters and corresponding motion trajectory of the operation object;

[0093] Based on the state characteristic data of the operation object, predicting a first operation action of the target robotic arm when performing the operation task, the first operation action including a movement trajectory from the end of the target robotic arm to the operation object and a contact position of the target robotic arm with the operation object;

[0094] Based on the first operation action, determining a first operation difficulty of the operation object relative to the operation type;

[0095] Determining a first operation influence of the operation object on the posture key point based on the first operation action and the first operation difficulty;

[0096] Based on the operation relevance and the first operation influence, a first efficiency evaluation value of each of the posture key points is determined, so that the posture key point corresponding to when the first efficiency evaluation value reaches a preset threshold is screened out as the target key point.

[0097] In this embodiment, the operation relevance of each posture key point is determined by the correlation between the operation type of the target manipulator and the posture key point, that is, the posture key point of the manipulator that affects the execution of the operation task, taking into account the degree of influence of the operation type on the posture key point. For example, the operation type is welding, and the manipulator needs to move up and down. At this time, the joint points on the manipulator involved in controlling the up and down movement have a high correlation with the operation type of the operation task, and the joint points on the manipulator involved in other movements such as horizontal movement and rotational movement have a low correlation with the operation task.

[0098] The operation image for the operation object is obtained by a binocular camera device to identify the state feature data of the operation object. Based on the state feature data, the operation action of the manipulator is predicted, for example, according to the posture, position and motion parameters (such as speed, acceleration, etc.) of the operation object, the corresponding operation action is matched in the database to obtain the movement trajectory of the manipulator that performs the operation task. Optionally, the operation object in this embodiment can be stationary, then the movement trajectory of the manipulator and the contact position of the operation object are quickly generated according to the position distance between the manipulator and the operation object, the posture of the operation object, etc. The operation object can also be moving, such as the operation object is located on a conveyor belt running at any speed, and the manipulator is used to perform operations such as grabbing the operation object, then the movement trajectory of the manipulator that performs the operation task and the contact position (such as the grabbing position) of the operation object can be determined according to the motion parameters of the operation object (such as the speed, acceleration, etc. of the operation object determined by the motion of the conveyor belt), posture, position, etc. Then, based on the operation action, the operation difficulty relative to the operation type is determined, and the first operation influence of the operation object on the posture key point is further calculated. For example, whether the operation object is moving, the motion parameters of the operation object, the position distance between the operation object and the robot arm, the size and posture of the operation object and other factors have different influence degrees on the key points.

[0099] Therefore, by considering the operation relevance of each posture key point on the operation type and the first operation influence on the operation object, the first efficiency evaluation value of the posture key point is calculated, so as to select the key point with a higher first efficiency evaluation value as the target key point. In this way, the number of key points to be calculated later is reduced, and there is no need to analyze all key points, which improves the key point detection speed. At the same time, it can meet the execution requirements of the current operation task and ensure that the target key point data has a high accuracy.

[0100] Based on the above embodiment, in another embodiment, the environmental data includes obstacles that affect the target manipulator to perform the operation task, and according to the operation task and environmental data of the target manipulator, the target key points corresponding to the posture key points that meet the preset screening conditions are screened, including:

[0101] Acquire an environment image with the obstacle sent by the binocular camera device, and determine obstacle feature data of the obstacle based on the environment image, wherein the obstacle feature data includes an obstacle position, size data, and motion state between the obstacle and the target robotic arm, and the motion state includes an obstacle motion trajectory corresponding to when the obstacle is a non-stationary obstacle;

[0102] Based on the obstacle position data and / or the motion state, modify the first operation action to obtain a second operation action;

[0103] Determining a first obstacle influence degree and a second obstacle influence degree based on the second operation action and the obstacle characteristic data, wherein the first obstacle influence degree is used to indicate the influence degree value of the obstacle on the execution of the operation type, and the second obstacle influence degree is used to characterize the motion correlation between the obstacle and the operation object;

[0104] Determining a second operation difficulty for performing the operation task when facing the obstacle based on the first obstacle impact degree, the second obstacle impact degree, and the first operation difficulty;

[0105] Determining a second operation influence of the operation object on the posture key point based on the second operation action and the second operation difficulty;

[0106] Determining an obstacle avoidance correlation between the posture key point and the obstacle based on the obstacle feature data;

[0107] Based on the operation relevance, the second operation influence and the obstacle avoidance relevance, the second efficiency evaluation value of each of the posture key points is determined, so that the posture key point corresponding to the second efficiency evaluation value reaching the preset threshold is screened out as the target key point.

[0108] It should be noted that in the scenario where multiple robotic arms work together, multiple robotic arms need to coordinate with each other to perform the operation task. How to make the robotic arms avoid possible interference and collision while completing the task quickly and accurately, which puts higher requirements on the real-time control of the motion trajectory of the robotic arms and the obstacle avoidance path planning. Therefore, this embodiment considers the environmental data of the robotic arm (such as the situation of obstacles) and the state characteristic data of the operation object, and evaluates the efficiency of the posture key points to screen out the key points that have a greater impact on the execution of the current operation task as the target key points, thereby fully considering the mutual influence and correlation between the environmental data, the state characteristic data of the operation object and the operation task, and screening out the key points that are suitable for the environmental data, state characteristic data and operation tasks, simplifying the subsequent calculation of the key points, improving the efficiency of key point detection, and ensuring the information richness and accuracy of the key points.

[0109] In this embodiment, an environment image containing obstacles is obtained through a binocular camera device to identify obstacle feature data, such as obstacle position, distance, size, motion state, etc. The motion state includes a stationary state and a non-stationary state (such as the motion trajectory of the obstacle in a non-stationary state). Then, the first operation action is corrected in combination with the obstacle feature data, the state feature data of the operation object, and the operation type to obtain a second operation action, such as the movement trajectory of the robot arm that performs the operation task when facing an obstacle and the contact position between the end of the robot arm and the operation object.

[0110] Further, based on the second operation action and the obstacle feature data, the first obstacle influence and the second obstacle influence are determined. The first obstacle influence is used to characterize the degree of influence of the obstacle on the execution of the operation task, and the second obstacle influence is used to characterize the correlation between the obstacle and the operation object. For example, the movement of the obstacle and the movement of the operation object are compared to determine the corresponding movement correlation, and the movement law and correlation between the obstacle and the operation object are taken into account to obtain the degree of influence of the obstacle on the end of the manipulator to the operation object. Thus, based on the first obstacle influence, the second obstacle influence and the first operation difficulty, the second operation difficulty when facing the obstacle is determined. Then, based on the second operation action and the second operation difficulty, the second operation influence of the operation object and the obstacle on the posture key point is determined, such as whether the obstacle moves, the movement parameters of the obstacle, the position distance between the obstacle and the operation object, whether the operation object moves, the movement parameters of the operation object, the position distance between the operation object and the manipulator, the size and posture of the operation object, etc., have different influences on the key points.

[0111] In addition, based on the obstacle feature data, the obstacle avoidance correlation between the obstacle and the posture key points is determined, that is, the posture key points of the robot arm that face the obstacle and perform the operation task. For example, according to the size of the obstacle, the correlation degree of each posture key point with avoiding the obstacle, that is, the obstacle avoidance correlation, takes into account the degree of influence of the obstacle on the posture key point.

[0112] Furthermore, by considering the operation relevance of each posture key point to the operation type, the second operation influence, and the obstacle avoidance relevance to the obstacle, the second efficiency evaluation value of the posture key point is calculated, thereby selecting the key point with a higher second efficiency evaluation value as the target key point. In this way, the number of key points to be calculated later is reduced, and there is no need to analyze all key points, which improves the key point detection speed. At the same time, it can meet the execution requirements of the current operation task and ensure that the target key point data has a high accuracy.

[0113] In the present application, if any camera device loses a key point during the detection process, the three-dimensional coordinates of the key point can be obtained by hole filling.

[0114] In this regard, in some embodiments, step S301 further includes:

[0115] When it is detected based on the key point detection model that any of the target key points is missing, obtaining a first task action in the first operation action at the current moment;

[0116] Obtaining a first moving position of the operation object at the current moment;

[0117] Determining at least one first associated target key point based on the first task action, the first moving position, the operation relevance, and the first operation influence;

[0118] Based on the three-dimensional coordinates of the first associated target key point, the first key point association degree and the key point trajectory, the three-dimensional coordinates of the missing target key point are predicted, and the first key point association degree is used to characterize the correlation between the first associated target key point and the missing target key point.

[0119] In the present embodiment, the first task action may be the action of the moving track of the first operation action at the current moment, thereby determining the corresponding moving position. Then, based on the first task action, the first moving position, the operation correlation and the first operation influence, one or more target key points are obtained from the currently detected key points as the first associated target key points, i.e., the target key points that affect the execution of the first task action. Then, based on the three-dimensional position of the first associated target key point, the correlation between the key points and the key point trajectory, the three-dimensional coordinates of the missing target key point are predicted. Therefore, the present embodiment combines the situation of performing the operation task (such as the correlation of the key point operation and the first operation influence related to the operation object) to determine the task action at the moment when the key point is missing, so as to associate the corresponding key points. Then, based on the three-dimensional position of the associated target key point, the correlation between the key points and the key point trajectory, the position of the missing target key point is predicted, thereby improving the prediction accuracy.

[0120] In some embodiments, step S301 further includes:

[0121] When it is detected based on the key point detection model that any of the target key points is missing, obtaining a second task action in the second operation action at the current moment;

[0122] Obtaining a second moving position of the operation object at the current moment;

[0123] Determining at least one second associated target key point based on the second task action, the second moving position, the operation relevance, the second operation influence, and the obstacle avoidance relevance;

[0124] Based on the three-dimensional coordinates of the second associated target key point, the second key point association degree and the key point trajectory, the three-dimensional coordinates of the missing target key point are predicted, and the second key point association degree is used to characterize the correlation between the second associated target key point and the missing target key point.

[0125] In this embodiment, the second task action can be the action of the moving trajectory of the second operation action at the current moment, thereby determining the corresponding moving position. Then, based on the second task action, the second moving position, the operation relevance to the operation type, and the second operation influence related to the operation object and the obstacle, one or more target key points are obtained from the currently detected key points as the second associated target key points, that is, the target key points that affect the execution of the second task action. Then, based on the three-dimensional position of the second associated target key points, the correlation between the key points, and the key point trajectory, the three-dimensional coordinates of the missing target key points are predicted.

[0126] Therefore, this embodiment combines the situation of executing the operation task (such as the relevance of the key point operation and the second operation influence related to the operation object and obstacle) to determine the task action at the moment when the key point is missing, so as to associate the corresponding key point. Then, according to the three-dimensional position of the associated target key point, the correlation between the key points and the key point trajectory, the position of the missing target key point is predicted, thereby improving the prediction accuracy.

[0127] Figure 5 The schematic diagram of the structure of a robot arm motion trajectory detection system provided in an embodiment of the present application, the robot arm motion trajectory detection system 500 comprises: a binocular camera device 501 and a processing device 502:

[0128] The binocular camera device 501 is used to obtain the image to be processed with the target robot arm, and then send it to the processing device;

[0129] The processing device 502 is connected to the binocular camera device 501 and is used to obtain the image to be processed;

[0130] Using a pre-established key point detection model to detect the image to be processed, and obtaining at least one posture key point and its corresponding two-dimensional coordinates;

[0131] Using a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory;

[0132] Performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates;

[0133] The target motion trajectory is compared with a preset motion trajectory, and a trajectory analysis result indicating whether the target robotic arm is abnormal is obtained according to the comparison result.

[0134] The system of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each device in the system of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each device of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0135] The present application also provides a computer program product, including a computer program, which implements the above-mentioned robot arm motion trajectory detection method when executed by a processor.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting the motion trajectory of a robotic arm, characterized in that: For a processing device, the method comprises: Acquire an image to be processed having a target robotic arm sent by a binocular camera device; Using a pre-established key point detection model to detect the image to be processed, and obtaining at least one posture key point and its corresponding two-dimensional coordinates; Using a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory; Performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates; The target motion trajectory is compared with a preset motion trajectory, and a trajectory analysis result indicating whether the target robotic arm is abnormal is obtained according to the comparison result.

2. The method for detecting the motion trajectory of a robot arm according to claim 1, characterized in that: The key point detection model is constructed through the following steps, including: Acquire a training data set, mark a bounding box of image training data in the training data set and key point positions of corresponding key points, wherein the key point includes at least one joint point on the target robotic arm; Get the preset target detection model; Constructing a target detection loss function based on the labeled training data set, wherein the target detection loss function is used to indicate training the bounding box; Based on the labeled training data set and the preset OKS indicator, a first key point loss function and a second key point loss function are constructed, wherein the first key point loss function is used to indicate the evaluation of the key point by the training, and the second key point loss function is used to indicate the confidence of the key point in the training; Based on the labeled training data set and the target detection loss function, construct a sorting and classification loss function, wherein the sorting and classification loss function is used to indicate the sorting and classification of samples in the training bounding box; Determine a model loss function based on the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function; The target detection model is trained based on the model loss function to obtain a key point detection model.

3. The method for detecting the motion trajectory of a robot arm according to claim 2, characterized in that: The determining of the model loss function based on the target detection loss function, the first key point loss function, the second key point loss function and the sorting classification loss function includes: The model loss function is determined by the following formula (1): L=∑ s,i,j,k (L RS +λ cls L cls +λ RS_box L box +λ RS_box L kpts_pos +λ RS_box L kpts_conf ) (1); Wherein, L is the model loss function, s represents the preset scale factor, i represents the first position parameter in the position (i, j), j represents the second position parameter in the position (i, j), k represents the instance, and L RS is the ranking classification loss function, L cls is the pre-acquired classification loss function, L box is the target detection loss function, L kpts_pos is the first key point loss function, L kpts_conf is the second key point loss function, λ cls is the first loss function hyperparameter about the classification loss function, λ RS_box is the second loss function hyperparameter; The second loss function hyperparameter is determined by the following formula (2): l RS_box =|L RS / L box | (2)。 4. The method for detecting the motion trajectory of a robot arm according to claim 1, characterized in that: The performing of parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates includes: According to the operation task and / or environmental data of the target manipulator, the target key points corresponding to the posture key points on the key point trajectory when they meet the preset screening conditions are screened, wherein the operation task includes the operation type and the state characteristic data of the operation object; Calculating the disparity of the target key point in the image to be processed; The target key point is triangulated based on the disparity to obtain the three-dimensional coordinates of the target key point.

5. The method for detecting the motion trajectory of a robot arm according to claim 4, characterized in that: The step of screening the target key points corresponding to the posture key points on the key point trajectory when they meet the preset screening conditions according to the operation task and / or environmental data of the target manipulator comprises: Based on the operation type of the target robotic arm, determining the operation correlation between the posture key point and the operation type; Acquire an operation image having the operation object sent by the binocular camera device, so as to identify state feature data of the operation object based on the operation image, wherein the state feature data includes posture data, target position data, motion parameters and corresponding motion trajectory of the operation object; Based on the state characteristic data of the operation object, predicting a first operation action of the target robotic arm when performing the operation task, the first operation action including a movement trajectory from the end of the target robotic arm to the operation object and a contact position of the target robotic arm with the operation object; Based on the first operation action, determining a first operation difficulty of the operation object relative to the operation type; Determining a first operation influence of the operation object on the posture key point based on the first operation action and the first operation difficulty; Based on the operation relevance and the first operation influence, a first efficiency evaluation value of each of the posture key points is determined, so that the posture key point corresponding to when the first efficiency evaluation value reaches a preset threshold is screened out as the target key point.

6. The method for detecting the motion trajectory of a robot arm according to claim 5, characterized in that: The environmental data includes obstacles that affect the target manipulator to perform the operation task; the target key points corresponding to the posture key points on the key point trajectory that meet the preset screening conditions are screened according to the operation task and / or environmental data of the target manipulator, and also include; Acquire an environment image with the obstacle sent by the binocular camera device, and determine obstacle feature data of the obstacle based on the environment image, wherein the obstacle feature data includes an obstacle position, size data, and motion state between the obstacle and the target robotic arm, and the motion state includes an obstacle motion trajectory corresponding to when the obstacle is a non-stationary obstacle; Based on the obstacle position data and / or the motion state, modify the first operation action to obtain a second operation action; Determining a first obstacle influence degree and a second obstacle influence degree based on the second operation action and the obstacle characteristic data, wherein the first obstacle influence degree is used to indicate the influence degree value of the obstacle on the execution of the operation type, and the second obstacle influence degree is used to characterize the motion correlation between the obstacle and the operation object; Determining a second operation difficulty for performing the operation task when facing the obstacle based on the first obstacle impact degree, the second obstacle impact degree, and the first operation difficulty; Determining a second operation influence of the operation object on the posture key point based on the second operation action and the second operation difficulty; Determining an obstacle avoidance correlation between the posture key point and the obstacle based on the obstacle feature data; Based on the operation relevance, the second operation influence and the obstacle avoidance relevance, the second efficiency evaluation value of each of the posture key points is determined, so that the posture key point corresponding to the second efficiency evaluation value reaching the preset threshold is screened out as the target key point.

7. The method for detecting the motion trajectory of a robot arm according to claim 6, characterized in that: The performing of parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates includes: When it is detected based on the key point detection model that any of the target key points is missing, obtaining a first task action in the first operation action at the current moment; Obtaining a first moving position of the operation object at the current moment; Determining at least one first associated target key point based on the first task action, the first moving position, the operation relevance, and the first operation influence; Based on the three-dimensional coordinates of the first associated target key point, the first key point association degree and the key point trajectory, the three-dimensional coordinates of the missing target key point are predicted, and the first key point association degree is used to characterize the correlation between the first associated target key point and the missing target key point.

8. The method for detecting the motion trajectory of a robot arm according to claim 6, characterized in that: The method further comprises: performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates; When it is detected based on the key point detection model that any of the target key points is missing, obtaining a second task action in the second operation action at the current moment; Obtaining a second moving position of the operation object at the current moment; Determining at least one second associated target key point based on the second task action, the second moving position, the operation relevance, the second operation influence, and the obstacle avoidance relevance; Based on the three-dimensional coordinates of the second associated target key point, the second key point association degree and the key point trajectory, the three-dimensional coordinates of the missing target key point are predicted, and the second key point association degree is used to characterize the correlation between the second associated target key point and the missing target key point.

9. A robot arm motion trajectory detection system, characterized in that: include: Binocular camera device and processing device: The binocular camera device is used to obtain the image to be processed with the target robotic arm, and then send it to the processing device; The processing device is connected to the binocular camera device and is used to obtain the image to be processed; Using a pre-established key point detection model to detect the image to be processed, and obtaining at least one posture key point and its corresponding two-dimensional coordinates; Using a preset filtering model to smooth the trajectory of the two-dimensional coordinates to obtain a key point trajectory; Performing parallax processing and triangulation on the posture key points on the key point trajectory to obtain the three-dimensional coordinates of the posture key points and the target motion trajectory including the three-dimensional coordinates; The target motion trajectory is compared with a preset motion trajectory, and a trajectory analysis result indicating whether the target robotic arm is abnormal is obtained according to the comparison result.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the robot arm motion trajectory detection method according to any one of claims 1 to 8 is implemented.

Images