Acquisition method and device of motion trail of mechanical arm and anomaly detection method and device of motion trail of mechanical arm

Abstract

The invention discloses a method and a device for acquiring the motion trail of a mechanical arm and a method and a device for detecting the anomaly of the mechanical arm, which can improve the acquisition efficiency of the motion trail of the mechanical arm by acquiring the motion trail of the mechanical arm through the method. The method for acquiring the motion trail of the mechanical arm comprises the steps of S100-S300. S100, an operation video of the mechanical arm is obtained; and S200, obtaining a data set of key points of the mechanical arm by using an OpenPose algorithm according to the operation video of the mechanical arm. And S300, according to the data set of the key points of the mechanical arm, the motion trails of the key points of the mechanical arm are obtained.

Description

Technical Field

[0001] The present invention specifically relates to a method and device for acquiring a motion trajectory of a robot arm and a method and device for detecting anomalies thereof. Background Art

[0002] Robotic arms are automated mechanical devices that are widely used in the field of robotics technology. They can be found in industrial manufacturing, medical treatment, entertainment services, semiconductor manufacturing, space exploration, etc. Although they have different forms, they all have one common feature, which is the ability to receive instructions and accurately locate a point in three-dimensional (or two-dimensional) space to perform operations.

[0003] During the use of the robotic arm, it is necessary to detect the movement trajectory of the robotic arm in real time, analyze and identify its behavior, and issue warnings for suspicious behaviors and behavior processes or determine whether they are compliant.

[0004] For the detection of robot arm motion trajectory, the current method mainly calculates the motion trajectory of the robot arm through forward and inverse kinematics. The steps of this method include: defining the robot arm model; calculating DH parameters; forward kinematics calculation; verifying forward kinematics; inverse kinematics calculation; verifying inverse kinematics; and evaluating experimental results. This method is simple and accurate, but when the robot arm program fails and the robot arm joint parameters are incorrect, the robot arm motion trajectory cannot be detected, which may lead to serious production accidents.

[0005] In recent years, with the continuous development of science and technology, machine vision has been gradually applied in transportation,

[0006] Medical, education and many other fields, as well as the recognition of robot arm movements. However, the existing machine vision recognition algorithms are not efficient in acquiring the robot arm's motion trajectory. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method and device for obtaining the motion trajectory of a robot arm and a method and device for detecting an abnormality thereof in view of the above-mentioned deficiencies in the prior art. By adopting this method to obtain the motion trajectory of the robot arm, the efficiency of obtaining the motion trajectory of the robot arm can be improved.

[0008] In a first aspect, an embodiment of the present invention provides a method for obtaining a motion trajectory of a robotic arm, and the method for obtaining a motion trajectory of a robotic arm comprises: steps S100-S300. Among them, S100, obtaining a running video of the robotic arm. S200, obtaining a data set of key points of the robotic arm using the OpenPose algorithm based on the running video of the robotic arm. S300, obtaining the motion trajectory of the key points of the robotic arm based on the data set of the key points of the robotic arm.

[0009] In some embodiments, the operation video of the robotic arm is an operation video of the robotic arm at multiple perspectives taken by multiple cameras located at different positions, and the step S100 specifically includes: calibrating multiple cameras to obtain intrinsic and extrinsic parameters of each camera; and obtaining the operation video of the robotic arm at multiple perspectives through the multiple cameras.

[0010] In some embodiments, the data set of the key points of the robot arm is a three-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives, and the step S200 includes: steps S210-S220. Among them, S210, based on the operation video of the robot arm under multiple perspectives, the key points of the robot arm are marked, and the internal and external parameters of each camera are combined to obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives using the OpenPose algorithm; S220, combining the internal and external parameters of each camera, the two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives is reconstructed in three dimensions to obtain a three-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

[0011] In some embodiments, after step S210 and before step S220, the method further includes: using a Kalman filtering method to correct a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives to obtain a complete two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

[0012] In some embodiments, step S210 includes: steps S211-S212. Among them, S211, data cleaning is performed on the operation videos of the robot arm under multiple viewing angles to obtain the cleaned operation videos; S212, key points of the robot arm are annotated according to the cleaned operation videos, and the two-dimensional key point data set based on the robot arm motion trajectory under multiple viewing angles is obtained by combining the internal and external parameters of each camera using the OpenPose algorithm.

[0013] In some embodiments, in step S212, the use of the OpenPose algorithm to obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives includes: steps S2121-S2128. S2121. Build an OpenPose network, initialize the connection layer weights, and define the stage level as n, where n is a positive integer greater than 1. S2122. Input any image in the operation video of the robot arm under multiple perspectives after data cleaning into CSPDarknet to extract image features and obtain an image feature atlas G corresponding to the image. S2123. Input the image feature atlas G into the first stage, and obtain the position estimation of the key points in the image and the vector set of key point connections through three layers of convolution kernels. S2124. Input the key point position estimation and the vector set of key point connections obtained in the previous stage and the image feature atlas G to the next stage to obtain the output of the next stage, and repeat the above process until the output of the nth stage is obtained. S2125. Repeat steps S2122-2124 until the output of the nth stage of all images is obtained, and then the target frame and key points of the robotic arm are obtained based on the output of the nth stage of all images. S2126. Calculate the loss function based on the target frame and key points of the robotic arm, and update the weight parameters of the OpenPose network through the back propagation algorithm based on the loss function. S2127. Repeat steps S2122-S2126, and obtain the OpenPose network model after multiple iterations. S2128. Input all images in the running video of the robotic arm from multiple perspectives after data cleaning into the OpenPose network model to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm from multiple perspectives.

[0014] In some embodiments, step S211 specifically includes: extracting the operation video of the robotic arm from multiple perspectives as pictures, calculating the blurriness of all pictures using the Laplace operator, and eliminating pictures with blurriness greater than a first threshold; calculating the similarity between each picture in the remaining pictures and other pictures using a structural similarity algorithm, and eliminating pictures with similarity greater than a second threshold to obtain a cleaned operation video.

[0015] Therefore, the method for acquiring the motion trajectory of the robotic arm provided by an embodiment of the present invention, after acquiring the operation video of the robotic arm, uses the OpenPose algorithm to obtain a data set of the key points of the robotic arm. Since the number of key points of the robotic arm is small and the detection scene is single, and the scene during operation is single and stable, the complexity of the key point detection of the robotic arm is reduced. When using the OpenPose algorithm to obtain the data set of the key points of the robotic arm, the number of output channels can be modified and the number of stages can be reduced, thereby improving the efficiency of acquiring the key points of the robotic arm and preventing overfitting, etc., and ultimately improving the efficiency of acquiring the motion trajectory of the robotic arm.

[0016] In a second aspect, an embodiment of the present invention further provides a method for detecting an abnormality in the motion trajectory of a robotic arm, the method comprising: using the method for acquiring the motion trajectory of the robotic arm in the first aspect to acquire the standard motion trajectory of the robotic arm; using the method for acquiring the motion trajectory of the robotic arm in the first aspect to acquire the real-time motion trajectory of the robotic arm; comparing the real-time motion trajectory of the robotic arm with the standard motion trajectory of the robotic arm, and judging whether the motion trajectory of the robotic arm is abnormal based on the comparison result.

[0017] In some embodiments, the real-time motion trajectory of the robotic arm is compared with the standard motion trajectory of the robotic arm to obtain a comparison result, specifically: the three-dimensional coordinates of each key point in the real-time motion trajectory of the robotic arm are compared with the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robotic arm to obtain the distance between the two.

[0018] In a third aspect, an embodiment of the present invention further provides a device for acquiring a motion trajectory of a robotic arm, the device comprising an acquisition component, a first computing module, and a second computing module. The acquisition component is used to acquire a running video of the robotic arm. The first computing module is connected to the acquisition component, and is used to obtain a data set of key points of the robotic arm using the OpenPose algorithm based on the running video of the robotic arm. The second computing module is connected to the first computing module, and is used to obtain the motion trajectory of the key points of the robotic arm based on the data set of the key points of the robotic arm.

[0019] In some embodiments, the acquisition component includes multiple cameras, which are arranged at different positions and are used to acquire the operation videos of the robot arm from multiple perspectives; the data set of the key points of the robot arm is a three-dimensional key point data set based on the robot arm motion trajectory from multiple perspectives. The first calculation module includes a two-dimensional key point data set calculation unit and a three-dimensional key point data set calculation unit. The two-dimensional key point data set calculation unit is connected to the multiple cameras respectively, and is used to mark the key points of the robot arm according to the operation videos of the robot arm from multiple perspectives, and to obtain the internal and external parameters of each camera, and combine the internal and external parameters of each camera to obtain the two-dimensional key point data set based on the robot arm motion trajectory from multiple perspectives using the OpenPose algorithm. The three-dimensional key point data set calculation unit is connected to the two-dimensional key point data set calculation unit, and is used to combine the internal and external parameters of each camera to perform three-dimensional reconstruction on the two-dimensional key point data set based on the robot arm motion trajectory from multiple perspectives, and obtain the three-dimensional key point data set based on the robot arm motion trajectory from multiple perspectives.

[0020] In a fourth aspect, an embodiment of the present invention further provides a device for detecting abnormal motion trajectory of a robot arm, and the device for detecting abnormal motion trajectory of a robot arm comprises the device for acquiring the motion trajectory of the robot arm and the comparison and judgment module in the third aspect. The device for acquiring the motion trajectory of the robot arm is used to acquire the standard motion trajectory of the robot arm and to acquire the real-time motion trajectory of the robot arm. The comparison and judgment module is connected to the device for acquiring the motion trajectory of the robot arm, and is used to compare the real-time motion trajectory of the robot arm with the standard motion trajectory of the robot arm, and to judge whether the robot arm has abnormal motion trajectory according to the comparison result.

[0021] The robot arm motion trajectory abnormality detection method, robot arm motion trajectory acquisition device, and robot arm motion trajectory abnormality detection device provided in the embodiments of the present invention have the same beneficial effects as the above-mentioned robot arm motion trajectory acquisition method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 : A flow chart of a method for obtaining a motion trajectory of a robotic arm provided by an embodiment of the present invention;

[0023] Figure 2 : A schematic diagram of a mechanical arm provided by an embodiment of the present invention;

[0024] Figure 3 : A flow chart of a method for detecting abnormal motion trajectory of a robotic arm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0026] Embodiment 1:

[0027] like Figure 1 As shown, an embodiment of the present invention provides a method for obtaining a motion trajectory of a robot arm, which is applied to a robot arm and used to detect the motion trajectory of the robot arm. The method for obtaining a motion trajectory of a robot arm includes steps S100-S300.

[0028] S100: Obtain an operation video of the robotic arm.

[0029] For example, Figure 2 As shown, the robotic arm may be a multi-joint robotic arm commonly used in the fields of industrial production, medical treatment, and the like.

[0030] In some examples, the operation video of the robotic arm is an operation video of the robotic arm from multiple perspectives taken by multiple cameras located at different positions. At this time, the above S100 specifically includes: calibrating multiple cameras to obtain the intrinsic parameters and extrinsic parameters of each camera; and obtaining the operation video of the robotic arm from multiple perspectives through multiple cameras.

[0031] Exemplarily, the number of the multiple cameras may be two or more. When the number of the multiple cameras is two, the two cameras are arranged on both sides of the robotic arm to respectively obtain the operation videos of the robotic arm from the left and right perspectives.

[0032] For example, a camera with higher resolution, higher frame rate and higher sensitivity can be selected. A high-resolution camera can provide clearer images to clearly capture the details of the robot arm, including the movement of its end effector and each joint, which helps to detect key points more accurately; a high-frame-rate camera can capture faster movements, especially when real-time monitoring and feedback control are required; a high-sensitivity camera can reduce the impact of factors such as light, occlusion, and environment on the final image.

[0033] In addition, the camera’s exposure time is required to be short to avoid blurring when capturing the robot’s motion, which is especially important for a robot that moves at high speeds.

[0034] The camera's contrast and color reproduction, lens and field of view, synchronization, stability, lighting conditions, interface and software support, and protective measures also require the selection of appropriate parameters.

[0035] Exemplarily, the intrinsic parameters of the camera include focal length and principal point coordinates, and the extrinsic parameters involve the position and posture of the camera relative to a known reference point.

[0036] Exemplarily, calibrating multiple cameras to obtain the intrinsic parameters and extrinsic parameters of each camera may include the following steps:

[0037] (1) Prepare a calibration plate: Use a calibration plate with known feature points, usually a black and white checkerboard or dot pattern. These feature points will be used to detect corresponding points in the image;

[0038] (2) Take calibration images: Place the calibration plate in front of the camera at different positions and angles and take multiple images. Make sure the calibration plate is fully visible in the image and that the image contains multiple viewpoints from close to long distances.

[0039] (3) Image detection: Use image processing algorithms to detect the corresponding positions of feature points on the calibration plate in the image. These algorithms can be feature-based detection methods, such as Harris corner detection or Shi-Tomasi corner detection;

[0040] (4) Establishing a coordinate system: Establish a three-dimensional coordinate system for the calibration plate, and correspond the detected feature point coordinates to the points in this coordinate system;

[0041] (5) Calibration algorithm: Select a calibration algorithm to calculate the camera parameters. Common algorithms include Zhang's method and the classic camera calibration method (using a linear equation system to solve);

[0042] (6) Calculate internal and external parameters: Calculate the camera's internal parameters (focal length, principal point, distortion coefficient, etc.) and external parameters (camera position and posture in the world coordinate system) through the calibration algorithm;

[0043] (7) Optimization and verification: The calibration results are optimized to reduce the error. Then, some verification methods are used to check the accuracy of the calibration, such as reprojection error.

[0044] The obtained intrinsic and extrinsic parameters of each camera can then be applied to actual visual tasks, such as 3D reconstruction described later.

[0045] S200, based on the operation video of the robotic arm, use the OpenPose algorithm to obtain a data set of key points of the robotic arm.

[0046] For a robotic arm, the movement of the joints determines the overall posture and position change of the robotic arm. Monitoring the joint position can accurately reflect the movement state of the robotic arm. Therefore, the position of the joints can be used as a key point when detecting the movement trajectory of the robotic arm.

[0047] OpenPose is an open source library based on convolutional neural networks and supervised learning and developed with Caffe as the framework. It can realize posture estimation of human movements, facial expressions, finger movements, etc. Since the number of key points of the robotic arm is greatly reduced compared to the human body (for example Figure 2 There are four key points of the robot arm, corresponding to Figure 2 The four points A, B, C, and D in the figure are total), and the scene at runtime is single and stable, and the complexity of detection is greatly reduced. Therefore, when using the OpenPose algorithm to obtain the data set of the key points of the robotic arm, the number of output channels can be modified and the number of stages can be reduced to improve efficiency and prevent overfitting, etc. It is more suitable for detecting the key points of the robotic arm, improves detection efficiency, and can handle complex scenes and situations where multiple robotic arms work at the same time.

[0048] S300: Obtain the motion trajectory of the key points of the robot arm according to the data set of the key points of the robot arm.

[0049] The method of obtaining the motion trajectory can be based on time series analysis. For example, the data set of the key points of the robot arm is arranged in chronological order to form a time series. For each key point, the motion trajectory of its position over time can be obtained.

[0050] Alternatively, you can also consider using time series analysis methods such as autoregressive model (AR), moving average model (MA), autoregressive moving average model (ARMA) or the more advanced autoregressive integrated moving average model (ARIMA).

[0051] Therefore, the method for acquiring the motion trajectory of a robotic arm provided by an embodiment of the present invention, after acquiring the operation video of the robotic arm, uses the OpenPose algorithm to obtain a data set of the key points of the robotic arm. Since the number of key points of the robotic arm is small and the detection scene is single, the complexity of detecting the key points of the robotic arm is reduced. When using the OpenPose algorithm to obtain the data set of the key points of the robotic arm, the number of output channels can be modified and the number of stages can be reduced, thereby improving efficiency and preventing overfitting, etc., and ultimately improving the efficiency of acquiring the motion trajectory of the key points of the robotic arm.

[0052] Moreover, because the acquisition method of the robot arm motion trajectory can reduce the complexity of key point detection of the robot arm, the method can be applied to complex scenes and situations where multiple robots work simultaneously. After testing, without losing accuracy, the overall detection efficiency of the robot arm motion trajectory is increased from the original 10fps to about 20fps. This method can complete the acquisition of the motion trajectory in a shorter time, meeting the needs of real-time monitoring.

[0053] In some embodiments, the above-mentioned data set of key points of the robotic arm is a three-dimensional key point data set based on the motion trajectory of the robotic arm under multiple perspectives, and step S200 includes: steps S210-S220.

[0054] S210. Based on the operation videos of the robotic arm from multiple perspectives, key points of the robotic arm are annotated, and the internal and external parameters of each camera are combined to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm from multiple perspectives using the OpenPose algorithm.

[0055] For example, the labelme annotation tool is used to annotate the key points of the robotic arm.

[0056] S220 , combining the internal parameters and external parameters of each camera, performing three-dimensional reconstruction on the two-dimensional key point dataset based on the motion trajectory of the robotic arm under multiple perspectives, and obtaining a three-dimensional key point dataset based on the motion trajectory of the robotic arm under multiple perspectives.

[0057] The step S220 may specifically include the following steps:

[0058] (1) Using SIFT or SURF, multiple pairs of same-name points (matching points) of two-dimensional key point data based on the robot arm motion trajectory under multiple perspectives are obtained;

[0059] (2) Obtain the initial intrinsic parameters of the camera obtained in the above steps;

[0060] (3) Based on the camera's intrinsic parameter matrix and matching pairs, the basic matrix F is calculated using the eight-point method + Ransac;

[0061] (4) Find the essential matrix E based on the basic matrix F and the internal parameter matrix;

[0062] (5) Based on the matching pairs of multiple viewpoints and camera parameters, triangulation is performed to obtain the coordinates of the 3D point, and the constraint that the 3D point is in front of the camera is used to select the correct camera poses R and T from four situations;

[0063] (6) Perform triangulation based on the matching pairs and camera parameters to obtain the three-dimensional coordinates of all matching pairs;

[0064] (7) Based on all three-dimensional coordinates, camera parameters and matching pair coordinates, the LM algorithm is used to perform bundle adjustment (nonlinear optimization) to obtain more accurate three-dimensional point coordinates and camera internal and external parameters.

[0065] In some embodiments, after step S210 and before step S220, the method further includes: using the Kalman filtering method to correct the two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives to obtain a complete two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

[0066] For example, since some joints of the robot arm may be blocked during movement, which makes it impossible to detect key points, it is necessary to use Kalman filtering to predict the key points of the blocked parts, and then obtain multiple groups of key points from different perspectives. The specific process of using the Kalman filtering method for correction may include the following steps:

[0067] (1) System modeling: Establish a dynamic model of the robot arm, including the state equation (describing the dynamic behavior of the system) and the observation equation (describing the relationship between observation and state). The state equation is usually expressed as: x_k = F_k*x_{k-1}+B_k*u_k+w_k, where x_k is the current state, x_{k-1} is the previous state, u_k is the control input, and w_k is the process noise. The observation equation is expressed as: z_k = H_k*x_k+v_k, where z_k is the observation value and v_k is the observation noise.

[0068] (2) Initialization: Initialize the state estimate x_0 and the estimated error covariance P_0 of the Kalman filter.

[0069] (3) Usually, the initial state estimation value can be set as the initial joint angle of the robot arm, and the initial estimation error covariance can be set based on experience or a randomly set initial value. In practical applications, the value of the initial estimation error covariance can also be determined by combining multiple methods (experimental method, method based on existing data, etc.). For example, the initial estimation error covariance can be set to 52=25.

[0070] (4) Prediction: Use the state equation to predict the estimated value of the next state: x_k^-=F_k*x_{k-1}^++B_k*u_k, where x_k^- is the prior estimate and x_{k-1}^+ is the posterior estimate of the previous state. Update the estimated error covariance: P_k^-=F_k*P_{k-1}^+*F_k^T+Q_k, where P_k^- is the prior estimate error covariance and Q_k is the process noise covariance.

[0071] (5) Update: When a new observation z_k arrives, calculate the Kalman gain: K_k = P_k^-*H_k^T*(H_k*P_k^-*H_k^T+R_k)^-1, where R_k is the observation noise covariance. Update the state estimate: x_k^+ = x_k^-+K_k*(z_k-H_k*x_k^-), where x_k^+ is the posterior estimate. Update the estimated error covariance: P_k^+ = (I-K_k*H_k)*P_k^-, where I is the identity matrix.

[0072] (6) Repeat step: Repeat the above prediction step and update step, using new observations to improve the state estimate in each iteration.

[0073] (7) Verification and adjustment: Verify the performance of the Kalman filter through actual measurement or simulation. Adjust model parameters such as the state transfer matrix F_k, control input matrix B_k, process noise Q_k, and observation noise R_k as needed to improve the filtering effect.

[0074] By using Kalman filtering for correction, the fluctuations that may be caused to key points by environmental interference can be removed.

[0075] In some embodiments, the above step S210 includes: steps S211-S212.

[0076] S211. Perform data cleaning on the operation videos of the robot arm from multiple viewing angles to obtain cleaned operation videos.

[0077] This step S211 specifically includes: extracting the operation videos of the robotic arm from multiple perspectives as pictures, using the Laplace operator to calculate the blurriness of all pictures, and eliminating pictures with blurriness greater than a first threshold; using a structural similarity algorithm to calculate the similarity between each picture in the remaining pictures and other pictures, and eliminating pictures with similarity greater than a second threshold, so as to obtain a cleaned operation video.

[0078] Exemplarily, the first threshold and the second threshold can be set as needed.

[0079] Exemplarily, images with blurriness greater than the first threshold may be blurred images caused by inaccurate camera focus, rapid movement of the robotic arm, or environmental factors. Removing such images ensures that the images used for subsequent key point detection and motion trajectory analysis have sufficient clarity, thereby improving the accuracy of detection.

[0080] For example, pictures with a similarity greater than the second threshold are pictures with too high similarity. Similar images may contain repeated information, which not only increases the amount of calculation, but also may cause deviations in key point detection results. Removing such pictures can reduce redundant data, optimize computing resource utilization, and avoid interference of similar images on subsequent analysis.

[0081] After data cleaning, data quality can be significantly improved, algorithm stability can be enhanced, and computing resource utilization can be optimized, which helps to more accurately obtain the motion trajectory of the robot arm and improve the performance and reliability of this method in practical applications.

[0082] S212. According to the running video after cleaning, the key points of the robot arm are marked, and the internal and external parameters of each camera are combined to obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives using the OpenPose algorithm.

[0083] In some embodiments, in step S212, the above-mentioned use of the OpenPose algorithm to obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives includes: steps S2121-S2128.

[0084] S2121. Build the OpenPose network, initialize the connection layer weights, and define the number of stages as n, where n is a positive integer greater than 1.

[0085] S2122. Input any image in the operation video of the robotic arm under multiple perspectives after data cleaning into CSPDarknet to extract image features and obtain an image feature atlas G corresponding to the image.

[0086] S2123. Input the image feature atlas G into the first stage, and obtain the position estimation of the key points in the image and the vector set of the key point connections through the three-layer convolution kernel.

[0087] S2124, input the key point position estimation obtained in the previous stage and the vector set of key point connections and the image feature atlas G to the next stage to obtain the output of the next stage, and repeat the above process until the output of the nth stage is obtained.

[0088] S2125. Repeat steps S2122-2124 until the output of the n-th stage of all images is obtained, and then obtain the target frame and key points of the robotic arm according to the output of the n-th stage of all images.

[0089] S2126. Calculate the loss function of the OpenPose network according to the target frame and key points of the robotic arm, and update the weight parameters of the OpenPose network through the back propagation algorithm according to the loss function.

[0090] S2127. Repeat steps S2122-S2126, and obtain the OpenPose network model after multiple rounds of iterations.

[0091] S2128. Input all images in the running video of the robotic arm from multiple perspectives after data cleaning into the OpenPose network model to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm from multiple perspectives.

[0092] That is, an OpenPose network model is obtained through the previous steps, and then the OpenPose network model is used to quickly obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

[0093] Embodiment 2:

[0094] The embodiment of the present invention also provides a method for detecting abnormal motion trajectory of a robot arm. The method is suitable for detecting a robot arm that performs repetitive motion, and can quickly identify abnormal motion trajectory of the robot arm to avoid production accidents caused by abnormal motion of the robot arm. The method for detecting abnormal motion trajectory of a robot arm comprises: using the method for acquiring the motion trajectory of a robot arm in embodiment 1 to acquire the standard motion trajectory of the robot arm; using the method for acquiring the motion trajectory of a robot arm in embodiment 1 to acquire the real-time motion trajectory of the robot arm; comparing the real-time motion trajectory of the robot arm with the standard motion trajectory of the robot arm, and judging whether the robot arm has abnormal motion trajectory according to the comparison result.

[0095] The flowchart of the robot arm motion trajectory anomaly detection method can be referred to Figure 3 .

[0096] Exemplarily, the standard motion trajectory of the robot arm is the motion trajectory of the robot arm when it operates normally, and the parameters of the motion trajectory can be stored after being obtained for subsequent use.

[0097] In some embodiments, the above-mentioned comparison of the real-time motion trajectory of the robotic arm with the standard motion trajectory of the robotic arm is specifically: comparing the three-dimensional coordinates of each key point in the real-time motion trajectory of the robotic arm with the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robotic arm to obtain the distance between the two.

[0098] Exemplarily, if the difference between the two is greater than a third threshold, it is determined that the robot arm has an abnormal motion trajectory; if the difference between the two is less than or equal to the third threshold, it is determined that the robot arm does not have an abnormal motion trajectory.

[0099] It can be understood that when the robotic arm is operating normally, the three-dimensional coordinates of the key points in the real-time motion trajectory of the robotic arm are no different or very close to the three-dimensional coordinates of the corresponding key points in the standard motion trajectory of the robotic arm; when the robotic arm is operating abnormally, the three-dimensional coordinates of the key points in the real-time motion trajectory of the robotic arm are significantly different from the three-dimensional coordinates of the corresponding key points in the standard motion trajectory of the robotic arm.

[0100] Exemplarily, the third threshold value can be set according to actual needs. When the motion trajectory of the robot arm requires a higher precision, the third threshold value can be set to a smaller value, for example, the third threshold value can be set to 1 mm. When the motion trajectory of the robot arm requires a lower precision, the third threshold value can be set to a larger value, for example, the third threshold value can be set to 3 mm.

[0101] Exemplarily, the above-mentioned third threshold is 1mm. At this time, the distance between the three-dimensional coordinates of each key point in the real-time motion trajectory of the robot arm and the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot arm is less than or equal to 1mm, which means that the robot arm is operating normally; when the distance between the three-dimensional coordinates of each key point in the real-time motion trajectory of the robot arm and the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot arm is greater than 1mm, the real-time motion trajectory of the robot arm deviates greatly from the standard motion trajectory of the robot arm, which means that the position or movement of the robot arm is abnormal, thereby judging that the robot arm has a trajectory abnormality.

[0102] In this embodiment, the motion trajectory of the robotic arm is obtained by the OpenPose algorithm, so the motion trajectory of the robotic arm is obtained quickly, and after comparison with the standard motion trajectory of the robotic arm, the trajectory abnormality of the robotic arm can be quickly detected.

[0103] Embodiment 3:

[0104] An embodiment of the present invention further provides a device for acquiring a motion trajectory of a robotic arm, the device comprising an acquisition component, a first computing module, and a second computing module. The acquisition component is used to acquire a running video of the robotic arm. The first computing module is connected to the acquisition component, and is used to obtain a data set of key points of the robotic arm using the OpenPose algorithm based on the running video of the robotic arm. The second computing module is connected to the first computing module, and is used to obtain the motion trajectory of the key points of the robotic arm based on the data set of the key points of the robotic arm.

[0105] Since the number of key points of the robotic arm is greatly reduced compared to the number of key points of the human body, the complexity of detection is greatly reduced. Therefore, when using the OpenPose algorithm to obtain a data set of the key points of the robotic arm, the number of output channels can be modified and the number of stages can be reduced to improve the efficiency of the first computing module in obtaining the data set of the key points of the robotic arm, so that the device for acquiring the motion trajectory of the robotic arm can quickly obtain the motion trajectory of the key points of the robotic arm.

[0106] In some examples, the acquisition component includes multiple cameras, which are arranged at different positions to acquire operation videos of the robot arm from multiple perspectives. The data set of key points of the robot arm is a three-dimensional key point data set based on the motion trajectory of the robot arm from multiple perspectives.

[0107] Exemplarily, the number of the multiple cameras may be two or more. The multiple cameras are arranged in different directions of the robotic arm to obtain operation videos of the robotic arm from multiple perspectives.

[0108] Exemplarily, the first computing module and the second computing module may both be general-purpose computers capable of performing data processing, and the first computing module and the second computing module both include a processor and a memory, etc.

[0109] The first calculation module includes a two-dimensional key point data set calculation unit and a three-dimensional key point data set calculation unit. The two-dimensional key point data set calculation unit is connected to multiple cameras respectively, and is used to mark the key points of the robot arm according to the operation video of the robot arm under multiple viewing angles, and to obtain the internal and external parameters of each camera, and combine the internal and external parameters of each camera to obtain the two-dimensional key point data set based on the robot arm motion trajectory under multiple viewing angles using the OpenPose algorithm. The three-dimensional key point data set calculation unit is connected to the two-dimensional key point data set calculation unit, and is used to combine the internal and external parameters of each camera to perform three-dimensional reconstruction on the two-dimensional key point data set based on the robot arm motion trajectory under multiple viewing angles, and obtain the three-dimensional key point data set based on the robot arm motion trajectory under multiple viewing angles.

[0110] Exemplarily, after acquiring the operation videos of the robotic arm from multiple perspectives, multiple cameras transmit the videos to the two-dimensional key point dataset calculation unit. The two-dimensional key point dataset calculation unit uses the OpenPose algorithm to obtain the two-dimensional key point dataset based on the motion trajectory of the robotic arm from multiple perspectives and transmits the obtained videos to the three-dimensional key point dataset calculation unit. The three-dimensional key point dataset calculation unit performs three-dimensional reconstruction on the two-dimensional key point dataset of the motion trajectory of the robotic arm, and finally obtains the three-dimensional key point dataset based on the motion trajectory of the robotic arm from multiple perspectives.

[0111] For example, the labelme annotation tool is used to annotate the key points of the robotic arm.

[0112] Thus, a three-dimensional key point data set based on the motion trajectory of the robot arm under multiple viewing angles can be obtained, so that the second calculation module can obtain the motion trajectory of the robot arm key points according to the three-dimensional key point data set based on the motion trajectory of the robot arm under multiple viewing angles.

[0113] Embodiment 4:

[0114] The embodiment of the present invention further provides a robot arm motion trajectory abnormality detection device, which is used to detect whether the robot arm has motion abnormality, and the robot arm motion trajectory abnormality detection device includes the robot arm motion trajectory acquisition device and the comparison and judgment module in Example 3. The robot arm motion trajectory acquisition device is used to acquire the standard motion trajectory of the robot arm and acquire the real-time motion trajectory of the robot arm. The comparison and judgment module is connected to the robot arm motion trajectory acquisition device, and is used to compare the real-time motion trajectory of the robot arm with the standard motion trajectory of the robot arm, and judge whether the robot arm has motion trajectory abnormality according to the comparison result.

[0115] Exemplarily, the comparison and judgment module may be a general-purpose computer capable of performing data processing, and the comparison and judgment module includes a processor and a memory, etc.

[0116] Exemplarily, the comparison result is: the distance between the three-dimensional coordinates of each key point in the real-time motion trajectory of the robot and the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot. If the distance is greater than the third threshold, it is determined that the robot has an abnormal motion trajectory; if the distance is less than or equal to the third threshold, it is determined that the robot does not have an abnormal motion trajectory.

[0117] Exemplarily, the third threshold is a preset distance.

[0118] For example, the third threshold is 1mm. At this time, if the distance between the three-dimensional coordinates of each key point in the real-time motion trajectory of the robot arm and the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot arm is greater than 1mm, the judgment result output by the comparison and judgment module is "abnormal", and the alarm component is driven to alarm; if the distance between the three-dimensional coordinates of each key point in the real-time motion trajectory of the robot arm and the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot arm is less than or equal to 1mm, the judgment result output by the comparison and judgment module is "normal".

[0119] Through the above settings, the real-time motion trajectory of the robotic arm can be detected by the robotic arm motion trajectory abnormality detection device, and the corresponding judgment result can be output when the real-time motion trajectory of the robotic arm is abnormal to remind the staff and avoid production accidents.

[0120] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for obtaining a motion trajectory of a robot arm, characterized in that: include: Steps S100-S300; S100, obtaining the operation video of the robotic arm; S200, obtaining a data set of key points of the robotic arm using an OpenPose algorithm according to the operation video of the robotic arm; S300: Obtain motion trajectories of the key points of the robotic arm according to a data set of the key points of the robotic arm.

2. The method for obtaining the motion trajectory of a robot arm according to claim 1, characterized in that: The operation video of the robotic arm is an operation video of the robotic arm under multiple viewing angles shot by multiple cameras disposed at different positions. The step S100 specifically includes: Calibrate multiple cameras to obtain the intrinsic and extrinsic parameters of each camera; The multiple cameras are used to obtain operation videos of the robotic arm from multiple perspectives.

3. The method for obtaining the motion trajectory of a robot arm according to claim 2, characterized in that: The data set of the key points of the robot arm is a three-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives, and the step S200 includes: steps S210-S220; S210, annotating key points of the robotic arm according to the operation videos of the robotic arm under multiple viewing angles, combining the intrinsic parameters and extrinsic parameters of each camera, and using the OpenPose algorithm to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm under multiple viewing angles; S220 , combining the internal parameters and external parameters of each camera, performing three-dimensional reconstruction on the two-dimensional key point dataset based on the motion trajectory of the robot arm under multiple perspectives, and obtaining a three-dimensional key point dataset based on the motion trajectory of the robot arm under multiple perspectives.

4. The method for obtaining the motion trajectory of a robot arm according to claim 3, characterized in that: After step S210 and before step S220, the method further includes: The two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives is corrected by using the Kalman filtering method to obtain a complete two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

5. The method for obtaining the motion trajectory of a robot arm according to claim 3, characterized in that: The step S210 includes: steps S211-S212; S211, performing data cleaning on the operation videos of the robot arm at multiple viewing angles to obtain cleaned operation videos; S212. According to the running video after cleaning, the key points of the robotic arm are marked, and the internal and external parameters of each camera are combined to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm under multiple perspectives using the OpenPose algorithm.

6. The method for obtaining the motion trajectory of a robot arm according to claim 5, characterized in that: In step S212, the method of using the OpenPose algorithm to obtain a two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives includes: steps S2121-S2128; S2121. Build the OpenPose network, initialize the connection layer weights, and define the number of stages as n, where n is a positive integer greater than 1; S2122, input any image in the running video after data cleaning into CSPDarknet to extract image features, and obtain an image feature atlas G corresponding to the image; S2123, input the image feature atlas G into the first stage, and obtain the position estimation of the key points in the image and the vector set of the key point connections through the three-layer convolution kernel; S2124, inputting the key point position estimation obtained in the previous stage and the vector set of key point connections and the image feature atlas G to the next stage to obtain the output of the next stage, and repeating the above process until the output of the nth stage is obtained; S2125, repeating steps S2122-2124 until the output of the n-th stage of all images is obtained, and then obtaining the target frame and key points of the robotic arm according to the output of the n-th stage of all images; S2126. Calculate the loss function of the OpenPose network according to the target frame and key points of the robotic arm, and update the weight parameters of the OpenPose network through the back propagation algorithm according to the loss function; S2127, repeat steps S2122-S2126, and obtain the OpenPose network model after multiple iterations; S2128. Input all images in the operation video of the robotic arm from multiple perspectives after data cleaning into the OpenPose network model to obtain a two-dimensional key point data set based on the motion trajectory of the robotic arm from multiple perspectives.

7. The method for obtaining the motion trajectory of a robot arm according to claim 5, characterized in that: Step S211 specifically includes: Extracting the operation videos of the robot arm from multiple perspectives as pictures, calculating the blurriness of all pictures using a Laplace operator, and eliminating pictures with blurriness greater than a first threshold; The structural similarity algorithm is used to calculate the similarity between each of the remaining pictures and the other pictures, and the pictures with similarity greater than the second threshold are eliminated to obtain the cleaned running video.

8. A method for detecting abnormal motion trajectory of a robot arm, characterized in that: include: Acquire the standard motion trajectory of the robotic arm using the method for acquiring the motion trajectory of the robotic arm as described in any one of claims 1 to 7; Acquire the real-time motion trajectory of the robotic arm using the method for acquiring the motion trajectory of the robotic arm as described in any one of claims 1 to 7; The real-time motion trajectory of the robotic arm is compared with the standard motion trajectory of the robotic arm, and it is determined whether the motion trajectory of the robotic arm is abnormal based on the comparison result.

9. The method for detecting abnormal motion trajectory of a robot arm according to claim 8, characterized in that: The real-time motion trajectory of the robot arm is compared with the standard motion trajectory of the robot arm, specifically: The three-dimensional coordinates of each key point in the real-time motion trajectory of the robot arm are compared with the three-dimensional coordinates of the corresponding key point in the standard motion trajectory of the robot arm to obtain the distance between the two.

10. A device for acquiring a motion trajectory of a robot arm, characterized in that: include: Acquisition component, used to obtain the operation video of the robotic arm; A first computing module, connected to the acquisition component, is used to obtain a data set of key points of the robotic arm using an OpenPose algorithm based on the operation video of the robotic arm; and, The second calculation module is connected to the first calculation module and is used to obtain the motion trajectory of the key points of the robotic arm according to the data set of the key points of the robotic arm.

11. The device for acquiring the motion trajectory of a robot arm according to claim 10, characterized in that: The acquisition component includes a plurality of cameras, which are arranged at different positions and are used to acquire operation videos of the robotic arm from multiple perspectives; The data set of the key points of the robot arm is a three-dimensional key point data set based on the motion trajectory of the robot arm under multiple viewing angles; The first calculation module includes: A two-dimensional key point data set calculation unit is connected to the multiple cameras respectively, and is used to mark the key points of the robotic arm according to the operation videos of the robotic arm under multiple viewing angles, and to obtain the internal and external parameters of each camera, and to obtain the two-dimensional key point data set based on the motion trajectory of the robotic arm under multiple viewing angles by using the OpenPose algorithm in combination with the internal and external parameters of each camera; and, The three-dimensional key point data set calculation unit is connected to the two-dimensional key point data set calculation unit and is used to combine the internal parameters and external parameters of each camera to perform three-dimensional reconstruction on the two-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives, so as to obtain the three-dimensional key point data set based on the robot arm motion trajectory under multiple perspectives.

12. A robot arm motion trajectory abnormality detection device, characterized in that: include: The device for acquiring the motion trajectory of a robotic arm as claimed in claim 10 or 11, used for acquiring the standard motion trajectory of the robotic arm and acquiring the real-time motion trajectory of the robotic arm; and, The comparison and judgment module is connected to the device for acquiring the motion trajectory of the robotic arm, and is used to compare the real-time motion trajectory of the robotic arm with the standard motion trajectory of the robotic arm, and judge whether the motion trajectory of the robotic arm is abnormal based on the comparison result.

Images