A digital-twin-based AI precise identification and grabbing method for picking and unhooking

Abstract

The present application relates to a kind of based on digital twinning's picking and hooking AI accurate identification and capture method, including creating the digital twin scene matched with real scene;In digital twin system, form a car type car hook model library;Through digital twin system, the accurate identification and capture of picking and hooking are carried out.The beneficial effects of the present application are: the present application obtains virtual data set for training by digital twin system, and is automatically labeled, to solve the problem of single fixed angle of traditional method, and the problem of low efficiency of artificial photographing and labeling of training data;And through data twinning system, identify the car type of rollover machine site, open car hook etc., determine its position relative to robot, replace the traditional laser positioning module and laser recognition module to find open car hook, so as to improve efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of car coupler uncoupling, more particularly, it relates to a digital twin-based AI precise identification and grabbing method for uncoupling and coupling hooks. BACKGROUND

[0002] The current technical development of the working robot in the site of the dumper is mainly to combine the grabbing device and the mechanical arm, to find the open car coupler by using the laser positioning module and the laser recognition module, and to determine the state and position of the coupler by using the image recognition technology, so as to perform the uncoupling and coupling operation. Since the environment around the dumper is harsh and the site is complex, the positioning detection accuracy and real-time performance of the robot in operation are required to be higher for the uncoupling and coupling operation. In the actual working process, the coal powder, environmental noise and electromagnetic noise in the site will have certain influence on the operation of the robot and the uncoupling precision. At present, there is no intelligent robot device that can properly solve the problem of automatic uncoupling and coupling of the dumper.

[0003] In order to determine the state and position of the coupler, in the image recognition model training, the site generally takes the RGB-D image as the input of the neural network to obtain the output of the grabbing posture rectangle box with direction. In this method, the camera of the mechanical arm is used to take pictures of the uncoupling and coupling in the site to obtain a large number of training RGB-D images in the early sampling stage, and the output is obtained. The mechanical arm is operated to perform the uncoupling and coupling operation. In this process, the camera has a relatively fixed and single shooting angle, which is generally on the side of the train. In this way, the car couplers with different shapes and poses are processed by fixed and single angle shooting, and the output obtained is relatively rough, resulting in a single uncoupling position and posture of the mechanical claw. At the same time, since a large amount of data is used for training the neural network, a large amount of time and effort will be spent on artificial photographing and labeling in the early stage. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings in the prior art, and to provide a digital twin-based AI precise identification and grabbing method for uncoupling and coupling hooks.

[0005] In the first aspect, a digital twin-based AI precise identification and grabbing method for uncoupling and coupling hooks is provided, comprising:

[0006] S1, creating a digital twin scene matched with the real scene;

[0007] S2, forming a car coupler model library in the digital twin system;

[0008] S3, determining the car type coming to the site by the digital twin system, and determining the site operation position that the robot should reach according to the real-time positions of the car and the robot in the digital twin system, and giving the optimized motion trajectory of the robot;

[0009] S4, generating a virtual dataset through a digital twin scene, and automatically labeling, the virtual dataset comprising multi-view RGB-D images;

[0010] S5, in a real scene, obtaining single-view RGB-D images according to a manual photographing method, thereby obtaining an actual dataset, and manually labeling;

[0011] S6, mixing the actual dataset and the virtual dataset in a one-to-many ratio to form a synthetic dataset;

[0012] S7, training an improved neural network model using the synthetic dataset as a training set, and evaluating the training results;

[0013] S8, using the trained neural network to predict the output pose to guide the gripper of the robot to grasp the decoupling hook;

[0014] S9, the digital twin system remotely controls the start or stop of the robot, while receiving the actual uploaded data on site, and performs background monitoring and data display.

[0015] Preferably, S4 comprises:

[0016] S401, for each decoupling hook, determining the activity radius of the robot and the grasping range, thereby determining the range of the graspable part of the decoupling hook;

[0017] S402, for the obtained graspable part range, setting a virtual camera view in the normal vector direction and moving it along the decoupling hook to obtain multi-view RGB-D images;

[0018] S403, automatically labeling the obtained virtual dataset according to the motion trajectory of the virtual camera and the relative object pose.

[0019] Preferably, in S403, first manually label the first data, then automatically obtain the subsequent label pose according to the previous label pose and motion trajectory, until all data in the virtual dataset are labeled.

[0020] Preferably, in S7, the improved neural network model comprises an object recognition and segmentation module and a pose estimation network module; in the pose estimation network module, the VGG16 network weight is used as the parameter of the convolution layer in the initialized pose estimation network; in the object recognition and segmentation module, BlendMask is used to extract the input information and perform RoI pooling; and after CNN convolution, the translation matrix T and the rotation matrix R are output; the object in the initial coordinate P A moves to the new coordinate P B under the translation matrix and the rotation matrix, and is represented as:

[0021]

[0022] As preferred, in S7, the training effect is evaluated using the average distance ADD, and the formula of ADD is:

[0023]

[0024] Where b is the number of points, B is the set of points, T and are the true value and predicted translation matrix, R and are the true value and predicted rotation matrix, and p is the coordinate (x, y, z) of the point in the three-dimensional model; if the calculated ADD is less than the preset threshold, the prediction result is considered correct.

[0025] As preferred, in S9, the uploaded data includes relative position and robot gripper state.

[0026] In a second aspect, a digital twin-based AI precise identification and grabbing system for uncoupling and recoupling hooks is provided, which is used to execute the digital twin-based AI precise identification and grabbing method for uncoupling and recoupling hooks according to any one of the first aspect, and comprises a digital twin system, a car body and a car hook, and a robot; wherein the digital twin system and the robot are in communication connection, and perform data acquisition and processing.

[0027] The beneficial effects of the present application are:

[0028] 1. The present application identifies the car type, open car hook, etc. of the incoming car on the car dumper site through the data twin system, and determines the position of the car relative to the robot, replacing the traditional use of laser positioning modules and laser recognition modules to find open car hooks, thereby improving efficiency and accuracy.

[0029] 2. The present application uses the digital twin system to obtain a virtual data set for training in multiple perspectives and automatically labels it to solve the problem of single fixed perspective in traditional methods, and the low efficiency of manual photographing and labeling of training data.

[0030] 3. The present application improves the object recognition and segmentation module in the image recognition neural network model, and uses the BlendMask module to replace the traditional Mask R-CNN module, etc. to make the proposals more accurate and fast.

[0031] 4. The present application uses the digital twin system to perform pre-rehearsal, manual intervention operation, etc. on the grabbing process, and records and analyzes the data of the grabbing failure process, etc. to improve the grabbing accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Fig. 1 is a structural schematic diagram of the digital twin-based AI precise identification and grabbing system for uncoupling and recoupling hooks;

[0033] Figure 2 A flowchart of a digital-twin-based unhooking and hooking AI precise recognition and grabbing method;

[0034] Figure 3 A structural diagram of an improved neural network model. DETAILED DESCRIPTION

[0035] The application will be further described below in conjunction with the embodiments. The following description of the embodiments is only to help understand the application. It should be noted that for those skilled in the art, without departing from the principles of the application, some modifications can be made to the application, and these improvements and modifications also fall within the scope of protection of the claims of the application.

[0036] Embodiment 1:

[0037] The application provides a digital-twin-based unhooking and hooking AI precise recognition and grabbing system, as shown in the figure, comprising a digital-twin system, a car body and a car hook, and a robot; wherein the digital-twin system and the robot are in communication connection and perform data acquisition and processing. Figure 1

[0038] Specifically, the digital-twin system is installed on a background host in a B / S architecture and interacts with real-world objects. Its main method is to map the real scene (mainly the car body, car hook and robot, etc.) into a three-dimensional virtual system, collect relevant data in the digital virtual system and process them, and finally feed back to the real robot for operation to complete the tasks that should be completed in the real scene to improve efficiency.

[0039] Embodiment 2:

[0040] The application provides a digital-twin-based unhooking and hooking AI precise recognition and grabbing method, comprising:

[0041] S1, creating a digital-twin scene matched with the real scene.

[0042] First, relying on the real scene, a high-precision matched digital-twin scene is created. The scene includes actual objects such as on-site incoming cars, car hooks, and on-site robots, and the modeling scale is required to be 1:1, and the mapping and rendering are required to be realistic. And accurately describe the intelligent scene process, and realize the data and information interaction between the real physical space and the digital-twin space.

[0043] S2, forming a car model hook model library in the digital-twin system.

[0044] In this way, when the incoming car model and the corresponding car hook change, the model library can directly replace the current running incoming car model and the corresponding car hook in the digital-twin system, so that repeated modeling is not needed.​

[0045] S3, determine the type of the incoming vehicle through the digital twin system, and determine the on-site operation position that the robot should reach according to the real-time positions of the vehicle compartment and the robot in the digital twin system, and give an optimized robot motion trajectory.

[0046] Compared with the traditional method of using a laser positioning module and a laser recognition module to find the car coupler, the efficiency and accuracy can be improved.

[0047] S4, generate a virtual data set through the digital twin scene, and automatically label it. The virtual data set includes multi-view RGB-D images.

[0048] S4 includes:

[0049] S401, for each uncoupling hook, determine the active radius of the robot and the grasping range, so as to determine the graspable part range of the uncoupling hook;

[0050] S402, for the obtained graspable part range, set the virtual camera view in the normal vector direction, and let it move along the uncoupling hook, so as to obtain multi-view RGB-D images;

[0051] S403, according to the motion trajectory of the virtual camera and the relative object pose, automatically label the obtained virtual data set. The moving path of the uncoupling hook in the digital twin scene and the photographing operation are automatically completed by the background code.

[0052] In S403, in order to avoid manually labeling all data, first manually label the pose in the first data, and then optimize the motion trajectory of the camera and the relative object pose on a global basis according to the virtual camera trajectory in the digital twin scene, so as to obtain a large amount of labeled data. Specifically, the camera moves from the previous position to the next position, which undergoes translation and rotation operations, from which the translation matrix and rotation matrix of the position transformation can be obtained. Since the camera is set in the direction of the normal vector of the uncoupling hook and uniformly moves along the uncoupling hook, the obtained camera translation matrix and rotation matrix can be directly applied to the transformation of the front and rear poses, so that the rear labeled pose can be automatically obtained according to the front labeled pose. The data set obtained in this way is a virtual data set. This process can be automatically completed by the system, and the data set and the labeling format can be freely set, thereby greatly improving the efficiency. In addition, since the virtual camera moves along the uncoupling hook in the digital twin three-dimensional scene, compared with the single photographing on one side of the train in the real scene, the view is more diverse. Therefore, multi-view virtual data is obtained in the digital twin scene and is automatically labeled, so as to solve the problem of single view of the data set and time-consuming and laborious photographing labeling in the prior art.

[0053] S5, in the real scene, according to the artificial photographing method, a single view RGB-D image is obtained, thus the actual data set is obtained, and manual annotation is carried out. It should be noted that the data amount in the actual data set is much smaller than that in the virtual data set.

[0054] S6, the actual data set and the virtual data set are mixed in a one-to-many ratio to form a synthetic data set. In order to improve the training effect, the samples are first randomized, and the sample size of each batch is designed to be 32 groups, and the learning rate is designed to be 0.001.

[0055] S7, the improved neural network model is trained by using the synthetic data set as the training set, and the training result is evaluated.

[0056] In S7, a visual neural network is constructed based on the Pose-RCNN network, and the improved neural network model includes an object recognition and segmentation module and a pose estimation network module. The prior art generally uses the VGG16 network weight trained on ImageNet as the parameter of the first 13 convolutional layers in the initialization pose estimation network, but subsequently uses Mask R-CNN algorithm to extract proposals and input RoI pool layer. The present application uses BlendMask to extract proposal information and input RoIPooling, thereby improving the accuracy and efficiency. Finally, the translation matrix T and the rotation matrix R are output after CNN convolution. The object in the initial coordinate P A is moved to the new coordinate P B after the translation matrix and the rotation matrix, that is, P B =RP A +T. Through the transformation formula , the object in the spatial coordinate P A is moved to the new position P B after the rotation matrix R and the translation T.

[0057] In addition, the model uses the average distance ADD to evaluate the training effect. The formula of ADD is as follows:

[0058]

[0059] Where b is the number of points, B is the set of points, T and are the true value and the predicted translation matrix, R and are the true value and the predicted rotation matrix, and p is the coordinate (x, y, z) of the point in the three-dimensional model. If the calculated ADD is less than the preset threshold, the prediction result is considered to be correct.

[0060] S8, the trained neural network is used to predict the output pose, and the output pose is transmitted to the mechanical arm to guide the gripper of the robot to grasp the hook.

[0061] S9, the digital twin system remotely controls the start or stop of the robot, while receiving the data actually uploaded on site, such as relative position, robot gripper state, etc., so as to perform background monitoring and data display, etc. Through the digital twin system, the grabbing process is preformed, artificial intervention operation, etc., or the data of the grabbing failure process is recorded and analyzed, etc., so as to improve the operability, especially for the automatic grabbing failure alarm, so that the artificial background operation robot completes the supplementary grabbing.

Claims

1. A digital twin-based AI precise identification and grabbing method for unhooking, characterized in that, Comprise: S1, create a digital twin scene matching the real scene; S2, form a car type coupler model library in the digital twin system; S3, determine the oncoming car type through the digital twin system, and determine the on-site operation position that the robot should reach according to the real-time position of the car body and the robot in the digital twin system, and give the optimized robot motion trajectory; S4, generate a virtual data set through a digital twin scene, and automatically label, the virtual data set comprising multi-view RGB-D images; S5, in the real scene, a single-view RGB-D image is obtained by artificial photographing method, thereby obtaining an actual data set, and artificial labeling is performed; S6, mix the actual data set and the virtual data set according to a one-to-many ratio to form a synthetic data set; S7, use the synthetic data set as a training set to train an improved neural network model, and evaluate the training result; In S7, the improved neural network model comprises an object recognition and segmentation module and a pose estimation network module; In the pose estimation network module, VGG16 network weights are used as the parameters of the convolutional layer in the initialized pose estimation network; in the object recognition and segmentation module, BlendMask is used to extract input information and perform RoI pooling; and after CNN convolution, a translation matrix T and a rotation matrix R are output; Object in initial coordinates P A After passing through the translation matrix and the rotation matrix, moves to new coordinates P B is represented as: In S7, the average distance ADD is used to evaluate the training effect, and the formula of ADD is: where b is the number of points, B is the set of points, T and are the true and predicted translation matrices, respectively, R and are the true and predicted rotation matrices, respectively, and p is the coordinate (x, y, z) of a point in the 3D model; the prediction is considered correct if the ADD is less than a pre-defined threshold value; S8, use the trained neural network to predict the output pose to guide the robot's gripper to grab the uncoupling hook; S9, the digital twin system remotely controls the start or stop of the robot, and simultaneously receives the uploaded data on site, performs background monitoring and data display.

2. The digital-twin-based AI accurate identification and unhooking method according to claim 1, wherein S4 Comprise: S401, for each uncoupling hook, determine the activity radius and the grabbing range of the robot, so as to determine the range of the uncoupling hook that can be grabbed; S402, for the obtained grabbable part range, set a virtual camera view angle in the normal vector direction, and let it move along the uncoupling hook, so as to obtain multi-view RGB-D images; S403, according to the motion trajectory of the virtual camera and the relative object pose, automatically label the obtained virtual data set.

3. The digital-twin-based AI accurate identification and picking hooking method according to claim 2, characterized in that, In S403, first manually label the first data, and then automatically obtain the subsequent labeling pose according to the previous labeling pose and motion trajectory, until all data in the virtual data set are labeled.

4. The digital-twin-based AI precise identification and grabbing method of claim 1, wherein In S9, the uploaded data includes relative position and robot gripper state.

5. A digital-twin-based AI precise identification and grabbing system for unhooking, characterized in that, The digital twin-based uncoupling hook AI precise identification and grabbing method for executing any one of claims 1 to 4, comprising: a digital twin system, a car body and a coupler, and a robot; wherein the digital twin system and the robot are in communication connection, and perform data acquisition and processing.

Images