A robot posture estimation method and system based on convolutional neural network

Abstract

The present invention relates to the field of neural network technology, and in particular to a robot posture estimation method and system based on convolutional neural network. The method comprises acquiring industrial robot posture information data; automatically labeling the acquired industrial robot posture information data to obtain a data set; preprocessing the data set; training a convolutional neural network model using the preprocessed data set; and estimating the posture of the robot using the trained convolutional neural network model. The present invention generates a high-precision industrial robot arm posture data set by combining automatic labeling with manual calibration, thereby overcoming the time-consuming and inaccurate manual labeling problems in traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a robot posture estimation method and system based on convolutional neural network. Background Art

[0002] With the continuous development of industrial automation, industrial robots have been widely used in manufacturing, assembly, packaging and other fields. These robots usually require accurate posture estimation to perform complex tasks, such as precise grasping, assembly or operating mechanical tools. In traditional methods, posture estimation usually relies on sensor-based data, such as lidar, camera or depth sensor. However, these methods usually require manual annotation of a large amount of data and face various challenges in practical applications, such as complex industrial environments, illumination changes, occlusion and sensor noise. Existing methods can solve the above problems to a certain extent, but still show the following shortcomings: 1. Existing methods usually perform well in specific scenarios or controlled environments, but have poor generalization ability in real industrial environments. The performance of the model often decreases significantly, making it difficult to cope with complex practical application requirements. 2. Traditional posture estimation methods rely on a large amount of manually annotated data, which not only requires a lot of manpower and time, but also easily introduces annotation errors, especially in complex industrial environments. 3. Industrial robots need to respond to environmental changes and task requirements in real time in practical applications, but existing methods usually have high computational complexity and slow processing speed, making it difficult to meet real-time requirements. 4. The robustness of existing methods is still insufficient when dealing with sensor noise, occlusion, and environmental changes.

[0003] At present, there are still the following defects in the posture estimation of robots and the acquisition of robot data sets: 1. Lack of data sets: Most of the existing data sets are collected in controlled environments, lacking data in actual industrial environments. These data sets in controlled environments usually cannot be generalized well to real-world applications. 2. High labeling cost: Since the posture estimation of robots requires accurate position information, the data labeling process is usually very complex and time-consuming. In real-world environments, manually labeling data is not only costly but also prone to errors. 3. Lack of diversity: Existing data sets may lack scene diversity, including different lighting conditions, complex backgrounds, and occlusions, which makes the trained models perform poorly in practical applications. 4. Poor real-time performance: In many industrial applications, robots need to process and respond in real time. However, existing methods may be insufficient in computational efficiency and difficult to meet the needs of real-time applications. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides a robot posture estimation method and system based on convolutional neural network.

[0005] In a first aspect, the present invention provides a robot posture estimation method based on a convolutional neural network, which adopts the following technical solution:

[0006] A robot posture estimation method based on convolutional neural network, comprising:

[0007] Obtaining industrial robot posture information data;

[0008] Automatically label the acquired industrial robot posture information data to obtain a data set;

[0009] Preprocess the dataset;

[0010] Use the preprocessed dataset to train the convolutional neural network model;

[0011] The trained convolutional neural network model is used to estimate the robot's posture.

[0012] Furthermore, the acquisition of the industrial robot posture information data includes capturing the motion posture of the industrial robot using a multi-camera and sensor system, wherein the motion trajectory of the robot is captured by the multi-camera system, the positions of multiple cameras are set, and the motion of the robot is captured from different angles at the same time, and the image sequence of each camera is represented as:

[0013]

[0014] in, Indicates The image captured by a camera at time t is and Represents the height and width of the image respectively, Indicates the number of channels. For RGB images, ,and is the number of cameras.

[0015] Furthermore, the acquisition of the industrial robot posture information data also includes using a joint encoder on the industrial robot to generate time series data of joint angles, and the joint angle data is expressed as:

[0016]

[0017] in, Indicates time Moment The angle of the joints, is the number of joints of the robot.

[0018] Furthermore, the acquired industrial robot posture information data is automatically labeled, including automatically generating accurate posture labels by fusing the industrial robot posture information data acquired by the sensor and the camera, and the posture labels correspond to the images to form training samples. , For the captured image, each joint angle of the industrial robot is used , calculate the position and orientation of the industrial robot end effector, expressed as:

[0019]

[0020] in, represents the transformation matrix of each joint, It means multiplying the transformation matrices of all joints to obtain the end position of the entire robot. When the posture of the end of the robot is represented by the displacement vector and the rotation matrix When expressed, the annotation formula is:

[0021]

[0022] in, It means the robot is in time The pose matrix of .

[0023] Furthermore, the preprocessing of the data set includes: Normalize the pixel values ​​to interval to meet the input requirements of the deep learning model. The standardized formula is:

[0024]

[0025] in, and represent the mean and standard deviation of the training data set respectively.

[0026] Furthermore, the use of the preprocessed data set to train the convolutional neural network model includes using the model to extract the posture features of the industrial robot from the 2D image of the data set, including key point features, wherein the coordinates of each joint of the industrial robot are , the 2D key points output by the convolutional neural network model are:

[0027]

[0028] Here, K is the number of joints of the robot.

[0029] Furthermore, the posture estimation of the robot using the trained convolutional neural network model includes evaluating the accuracy of the model in posture estimation by calculating the average key point error MKE, which is expressed as:

[0030]

[0031] in, is the total number of key points; The model predicts The two-dimensional coordinates of the key points are expressed as ( ); For the real The two-dimensional coordinates of the key points are expressed as ( ); is the Euclidean distance, which is used to calculate the distance between the predicted key points and the true key points.

[0032] In the second aspect, a robot posture estimation system based on a convolutional neural network comprises:

[0033] A data acquisition module is configured to acquire the posture information data of the industrial robot;

[0034] The labeling module is configured to automatically label the acquired industrial robot posture information data to obtain a data set;

[0035] The preprocessing module is configured to preprocess the data set;

[0036] The model training module is configured to train a convolutional neural network model using the preprocessed data set;

[0037] The model prediction module is configured to use the trained convolutional neural network model to estimate the posture of the robot.

[0038] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device for the robot posture estimation method based on a convolutional neural network.

[0039] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the robot posture estimation method based on a convolutional neural network.

[0040] In summary, the present invention has the following beneficial technical effects:

[0041] The present invention generates a high-precision industrial robot arm posture data set by combining automated labeling with manual calibration, overcoming the time-consuming and inaccurate manual labeling problems in traditional methods. Subsequently, manual calibration is used to correct sensor errors and inconsistencies in the visual system, further improving the labeling accuracy, the accuracy and robustness of posture estimation, and thus providing a more reliable training data source for the data-driven robot posture estimation model. Ultimately, a system that can collect and label robot arm posture data on a large scale and with high precision in a complex industrial environment is realized, and then a lightweight convolutional neural network is used to estimate the robot's posture, effectively improving the performance and generalization ability of the posture estimation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of a robot posture estimation method based on a convolutional neural network according to Example 1 of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] Example 1

[0045] Reference Figure 1 , a robot posture estimation method based on a convolutional neural network in this embodiment includes:

[0046] Obtaining industrial robot posture information data;

[0047] Automatically label the acquired industrial robot posture information data to obtain a data set;

[0048] Preprocess the dataset;

[0049] Use the preprocessed dataset to train the convolutional neural network model;

[0050] The trained convolutional neural network model is used to estimate the robot's posture.

[0051] Specifically:

[0052] S1: Data collection: In a real industrial environment, multiple cameras and sensor systems are used to capture the motion posture of industrial robots. Cameras are arranged to cover the robot's motion range at multiple angles to capture different posture information of the robot at different positions.

[0053] S2: Automated labeling: Combine visual and motion sensor data to automatically label the collected data set. The joint encoder provides the robot's internal motion data. By combining the robot's internal data with the external visual sensor data, the robot's joint position information and posture labels are generated. Manual calibration is then used to correct system errors, inconsistencies between camera and sensor data, and to address labeling issues in complex environments.

[0054] S3: Data preprocessing and enhancement: Perform data preprocessing steps on the data set, remove noise, standardize the image, align the image, etc. Perform data enhancement operations on the data set by performing random rotation, scaling, lighting changes, etc., expand the data set, and increase the generalization ability of the model.

[0055] S4: Neural network model training: The labeled and preprocessed dataset is used as input, and a lightweight convolutional neural network is used to train the robot posture data, so that the model can extract the robot's posture features from the 2D image and accurately estimate the robot's posture.

[0056] S5: Model Validation and Testing: Generate robot datasets through automatic data annotation and diversified datasets in multiple random and real industrial environments, and then use these datasets to estimate the robot's posture.

[0057] Preferably, a further technical solution of the present invention is:

[0058] S1 uses multiple cameras and sensor systems to capture the motion posture of industrial robots. The multi-camera system captures the motion trajectory of the robot. The positions of multiple cameras can be set so that they can capture the movement of the robot from different angles at the same time. The image sequence of each camera is represented as:

[0059]

[0060] in, Indicates The image captured by a camera at time t is and Represents the height and width of the image respectively, Indicates the number of channels (for RGB images, ),and is the number of cameras. At the same time, the joint encoders on the robot will generate time series data of joint angles, which can be expressed as:

[0061]

[0062] in, Indicates time Moment The angle of the joints, is the number of joints of the robot.

[0063] S2's automated annotation system combines visual and motion sensor data and uses forward kinematics (FK) to calculate the pose of the robot's end effector. , the position and orientation of the robot end effector can be calculated by the following formula:

[0064]

[0065] in, represents the transformation matrix of each joint, It means multiplying the transformation matrices of all joints to get the end position of the entire robot. If the posture of the robot end is represented by the displacement vector and the rotation matrix Indicates that the annotation formula is:

[0066]

[0067] in, It means the robot is in time By fusing sensor data and camera data, accurate pose labels can be automatically generated. These labels correspond to images and form training samples. , For the captured image.

[0068] S2 uses manual calibration to correct system errors, inconsistencies between camera and sensor data, and to deal with labeling problems in complex environments. The errors in automated labeling are corrected through the graphical user interface (manual labeling GUI) of the created calibration tool. The process is as follows:

[0069] 1. Image acquisition obtains the current image of the camera system, including the corresponding joint posture information of the robot, such as three-dimensional joint information.

[0070] 2. Mark the 2D joint positions of the robot. The manual annotation GUI supports the position annotation of the robot kinematic chain and allows the marked joint positions of the robot to be moved on the image.

[0071] 3. Calibrate based on the 2D position of the marker and the 3D joint correspondence obtained in the first step.

[0072] 4. Repeat steps 1 to 4 until the calibration is complete.

[0073] S3 image normalization, all input images Standardization is required to normalize the pixel values ​​to interval to meet the input requirements of the deep learning model. The standardized formula is:

[0074]

[0075] in, and represent the mean and standard deviation of the training data set respectively.

[0076] The S4 model can extract the robot's posture features from the 2D image. The CNN model outputs the robot's 2D posture estimate, which is the coordinates of the key points. The coordinates of each joint of the robot are , the 2D key points output by the model are:

[0077]

[0078] Where K is the number of joints of the robot. In order to optimize the model so that its estimated 2D pose is as close as possible to the true label, the Euclidean distance loss function is used:

[0079]

[0080] in, The model predicts The 2D coordinates of the joints, is the corresponding true value. Then use the Adam optimization algorithm to train the model, and the optimization goal is to minimize the loss function:

[0081]

[0082] in, Represents the parameters of the CNN model, and the Adam algorithm updates the parameters through gradient descent.

[0083] The S5 model validation process uses test data collected from real industrial environments to evaluate the performance of the model. The accuracy of the model in pose estimation is evaluated by calculating the mean keypoint error (MKE):

[0084]

[0085] in, is the total number of key points, The model predicts The two-dimensional coordinates of the key points. Usually expressed as . For the real The two-dimensional coordinates of the key points. Usually expressed as . is the Euclidean distance, which is used to calculate the distance between the predicted key points and the real key points. The performance of the model can be quantified by calculating the MKE value of the model on the test data set. A low MKE value indicates that the model estimates the robot posture more accurately under various conditions (such as complex lighting, occlusion, etc.), verifying the robustness and generalization ability of the model.

[0086] Example 2

[0087] This embodiment provides a robot posture estimation system based on a convolutional neural network, comprising:

[0088] A data acquisition module is configured to acquire the posture information data of the industrial robot;

[0089] The labeling module is configured to automatically label the acquired industrial robot posture information data to obtain a data set;

[0090] The preprocessing module is configured to preprocess the data set;

[0091] The model training module is configured to train a convolutional neural network model using the preprocessed data set;

[0092] The model prediction module is configured to use the trained convolutional neural network model to estimate the posture of the robot.

[0093] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for a robot posture estimation method based on a convolutional neural network.

[0094] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the robot posture estimation method based on a convolutional neural network.

[0095] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot posture estimation method based on convolutional neural network, characterized in that: include: Obtaining industrial robot posture information data; Automatically label the acquired industrial robot posture information data to obtain a data set; Preprocess the dataset; Use the preprocessed dataset to train the convolutional neural network model; Use the trained convolutional neural network model to estimate the robot's posture; The acquisition of the industrial robot posture information data includes capturing the motion posture of the industrial robot using a multi-camera and sensor system, wherein the motion trajectory of the robot is captured by the multi-camera system, the positions of multiple cameras are set, and the motion of the robot is captured from different angles at the same time, and the image sequence of each camera is represented as: in, Indicates The image captured by a camera at time t is and Represents the height and width of the image respectively, Indicates the number of channels. For RGB images, ,and is the number of cameras; The acquisition of the industrial robot posture information data also includes using the joint encoder on the industrial robot to generate time series data of the joint angle, and the joint angle data is expressed as: in, Indicates time Moment The angle of the joints, is the number of joints of the robot; The automatic labeling of the acquired industrial robot posture information data includes automatically generating accurate posture labels by fusing the industrial robot posture information data acquired by sensors and cameras. The posture labels correspond to the images to form training samples. , For the captured image, each joint angle of the industrial robot is used , calculate the position and orientation of the industrial robot end effector, expressed as: in, represents the transformation matrix of each joint, It means multiplying the transformation matrices of all joints to obtain the end position of the entire robot. When the posture of the end of the robot is represented by the displacement vector and the rotation matrix When expressed, the annotation formula is: in, It means the robot is in time The pose matrix of The data set is preprocessed, including the input image of the data set Normalize the pixel values ​​to interval to meet the input requirements of the deep learning model. The standardized formula is: in, and Represent the mean and standard deviation of the training data set respectively; The method of using the preprocessed data set to train the convolutional neural network model includes using the model to extract the posture features of the industrial robot from the 2D image of the data set, including key point features, wherein the coordinates of each joint of the industrial robot are , the 2D key points output by the convolutional neural network model are: Where K is the number of joints of the robot; The robot posture estimation is performed using the trained convolutional neural network model, including evaluating the accuracy of the model in posture estimation by calculating the average key point error MKE, which is expressed as: in, is the total number of key points; The model predicts The two-dimensional coordinates of the key points are expressed as ( ); For the real The two-dimensional coordinates of the key points are expressed as ( ); is the Euclidean distance, which is used to calculate the distance between the predicted key point and the true key point.

2. A robot posture estimation system based on convolutional neural network, executing a robot posture estimation method based on convolutional neural network as claimed in claim 1, characterized in that: include: A data acquisition module is configured to acquire the posture information data of the industrial robot; The labeling module is configured to automatically label the acquired industrial robot posture information data to obtain a data set; The preprocessing module is configured to preprocess the data set; The model training module is configured to train a convolutional neural network model using the preprocessed data set; The model prediction module is configured to use the trained convolutional neural network model to estimate the posture of the robot.

3. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

4. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images