An object pose prediction algorithm based on deep learning without depth information

Abstract

The present invention belongs to the field of image processing technology. The present invention discloses an object posture prediction algorithm based on deep learning without depth information, which can predict the category information and posture information of the object contained in the picture lacking depth information in real time, and can effectively reduce the target posture detection cost, and improve the real-time performance of target recognition and target posture detection. The object posture prediction algorithm based on deep learning without depth information described in the present invention only needs to use ordinary RGB images as input, directly predict the two-dimensional projection of the 3D bounding box vertices of the target object through a convolutional neural network, and calculate the posture information of the target object after obtaining the external parameters of the camera by solving the camera posture through the Aruco two-dimensional code. The average 2D projection accuracy of the posture prediction algorithm proposed in the present invention under the Linemod data set can reach 90.02%, which can effectively reduce the cost of target posture detection work, improve the efficiency of target detection, and ensure the real-time performance of target detection and posture prediction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an algorithm capable of accurately and in real time predicting category information and position information of objects contained in a picture without depth information. Background Art

[0002] With the rapid development of computer, multimedia and data communication technology, digital image processing has received great attention and rapid development in recent years, and has been widely used in scientific research, industrial production, medical care, education, entertainment, management and communication. At the same time, people's requirements for computer video applications are getting higher and higher, making high-performance digital image processing equipment that is fast, convenient and intelligent become the development direction of future video equipment. Real-time image processing technology has been increasingly used in target tracking, robot navigation and assisted driving. Due to the large amount of data in image processing, the high correlation of data processing, and the strict frame, field and time restrictions determined by the real-time application environment, real-time image processing must have powerful computing power.

[0003] In modern industrial automation production, there are many applications involving product inspection, production monitoring, and component defect identification. Binocular cameras are usually used to process image information. Compared with monocular cameras, binocular cameras are more expensive. Using binocular cameras will increase the cost of the robot grasping system. The power consumption of binocular cameras is much higher than that of monocular cameras. At the same time, in the use of industrial automation production robotic arms, the robotic arms are usually isolated from the surrounding environment to prevent robotic arm safety problems caused by unexpected situations. The use scene of the robotic arm must be open to ensure the safety of the robotic arm when working. At the same time, the grasping robotic arm also has problems such as poor grasping accuracy, low grasping accuracy, low grasping efficiency, and inability to autonomously avoid obstacles. Summary of the invention

[0004] The purpose of the present invention is to provide an object pose prediction algorithm based on deep learning without the need for depth information, which can predict the category information and pose information of objects contained in a picture without depth information.

[0005] The object pose prediction algorithm based on deep learning without depth information proposed in the present invention is a convolutional neural network architecture based on a monocular camera. Unlike the traditional pose estimation algorithm that requires matching RGB images with depth images, this algorithm does not require depth images, but only needs to use ordinary RGB images as input. The convolutional neural network directly predicts the two-dimensional projection of the 3D bounding box vertices of the target object, converts the pose estimation problem into a PnP solution problem, and the pose information of the target object can be calculated after the camera's external parameters are obtained by solving the camera pose through the Aruco QR code. The algorithm can predict the category information and pose information of objects contained in RGB images in real time, and its average 2D projection accuracy under the Linemod dataset can reach 90.02%.

[0006] The algorithm uses a residual network for feature extraction so that the network can extract deeper feature values. Since multiple residual networks are used in the feature extraction network, the network uses convolution instead of pooling to modify the size of the feature tensor, so as to ensure that the secondary related features in the feature tensor can be extracted instead of discarding them using the pooling layer. The network uses Darknet53 as the feature extraction network, and uses the PassThrough layer to modify the size of the tensor. The modified tensor is spliced ​​with the feature extraction network to expand the dimension, so that the network can obtain multi-dimensional features. In order to ensure that the center point of the feature map after downsampling extracts the features, the input of the convolutional neural network uses a 416×416×3 RGB image, so that the length and width of the output tensor of the feature extraction network are odd numbers. After the feature map is extracted, there is only one grid in the center of the feature map. After the input RGB image is downsampled 32 times, a tensor of 13×13×(19+C) dimensions is output.

[0007] The algorithm formulates the 6D pose estimation problem based on the 2D image coordinates of virtual 3D control points associated with the predicted 3D model of the object. Given the 2D coordinate predictions, the 6D pose of the object is calculated using the EPnP algorithm. The 3D model of each object is parameterized using 9 control points. For these control points, the 8 corners of a tight 3D bounding box are selected to fit the 3D model. In addition, the centroid of the 3D model of the object is used as the 9th point. The parameterization method is universal and can be used for 3D rigid objects of arbitrary shape and topology. These 9 control points are guaranteed to be well distributed in the 2D image.

[0008] Furthermore, the model takes a single full-color image as input, processes it with a fully convolutional structure, and divides the image into a 2D regular grid containing S×S cells. In the model, each grid position in the 3D output tensor will be associated with a multidimensional vector consisting of the predicted 2D image position of the 9 control points, the class probability of the object, and the overall confidence value. During training, it will be pruned where the predicted object does not exist. In the case of 2D object detection, the intersection of the IoU scores associated with the prediction in the image is used as the confidence value. The confidence value is expressed as:

[0009]

[0010] Where D(x) is the Euclidean distance between the predicted 2D point coordinate value and the true value, d th It is a threshold set in advance.

[0011] Furthermore, the last layer of the network outputs the class probabilities, the (x,y) coordinate locations of the control points, and the overall confidence scores. During training, the predicted two-dimensional coordinates are relative to (c x ,c y ), which is the upper left corner of the relevant grid cell. For the centroid, this offset is constrained between 0 and 1; for corner points, since these points may fall outside the cell, the output of the network is not constrained. Define the predicted control point (g x ,g y )for:

[0012] g x =f(x)+c x

[0013] g y =f(y)+c y

[0014] Where f() is a sigmoid function in the case of the center of mass and an identity function in the case of the eight corners. This will force the network to first find an approximate unit location for the target and then refine the locations of its eight corners. The loss function of the network is:

[0015]

[0016] Among them, the first item is the centroid coordinate loss of the predicted target frame, the second item is the coordinate loss of the target frame that is not responsible for predicting, the third item is the confidence loss of the cell responsible for predicting this target frame, the fourth item is the confidence loss of the cell that is not responsible for predicting the target frame, the sum of the third and fourth items is the predicted confidence loss, and the last item is the category loss of the predicted target frame.

[0017] Beneficial effects of the present invention:

[0018] The object pose prediction algorithm based on deep learning and without depth information has good real-time processing performance and high accuracy because the network structure used is the YOLO network. On the other hand, because the image acquisition device is a monocular camera, the use of a traditional binocular camera is avoided, which can reduce the hardware cost required for 6D pose recognition while ensuring recognition accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Network structure diagram of the object pose prediction algorithm

[0020] Figure 2 Structural diagram of the robotic arm control system to implement the algorithm

[0021] Figure 3 Object recognition flow chart for object pose prediction algorithm DETAILED DESCRIPTION

[0022] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0023] Reference Figure 1 The present invention provides an object pose prediction algorithm based on deep learning without depth information. The image input to the network is a conventional 416*416 RGB image. The network includes a CSPDarknet53 module, 35 CBM modules, 1 SPP module, 4 tensor splicing modules, 3 Conv modules and two upsampling modules.

[0024] Reference Figure 2 ,The structure diagram of the robotic arm control system that implements the algorithm, includes two systems: the visual system and the robotic arm system;

[0025] In this embodiment, the visual system includes a monocular camera, USB communication and a host computer; the robotic arm system includes a communication unit, a power supply unit, a control unit and an actuator unit;

[0026] In this embodiment, the monocular camera is responsible for collecting image information, and the USB communication is responsible for transmitting the image information collected by the monocular camera to the host computer. The host computer is responsible for predicting the object information contained in the image and identifying the object type and posture through the improved YOLO algorithm;

[0027] In this embodiment, the communication unit includes Ethernet communication, which transmits the data used by the host computer to control the position of the robotic arm to the control unit; the control unit includes a motion controller, which is essentially a programmable logic controller, which transmits the received control information to the drive module; the power supply unit includes a power supply and a power conversion module, the power supply is used to power the entire system, and the power conversion module is responsible for converting the voltage and providing it to the communication unit, the control unit and the actuator unit; the actuator unit includes a drive module, a stepper motor, a robotic arm, and an end effector. The drive module is responsible for receiving the data transmitted by the motion controller and driving the stepper motor and the end effector. The stepper motor is responsible for driving the robotic arm to move, and the end effector performs a grasping action according to the received information.

[0028] Reference Figure 3 , the object recognition flow chart of the object pose prediction algorithm provided by the present invention, firstly, the collected image information is transmitted to the deep learning network of the host computer through USB communication through the monocular camera, and after being processed by the deep learning network, the network outputs the particle position of the predicted object contained in the image and its two-dimensional projection, and then the image is transmitted to the YOLO6D network introduced into the 3D bounding box, and the network outputs the 8 corner points of the predicted object. At this time, we get the two-dimensional projection, 1 particle, and 8 corner points of the predicted object. Next, 4 non-coplanar points are selected for EPnP solution to obtain the three-dimensional coordinates of the predicted object. In order to obtain accurate results, it is then detected whether all control points have been calculated and used. If not all of them have been used, we re-select control points for calculation until all control points are used for EPnP solution. When all control points have been solved by EPnP, we get the 6D pose of the predicted object, which includes position information and pose information.

Claims

1. A method for predicting object pose based on deep learning without depth information, characterized in that: A method for object pose prediction based on deep learning without depth information is proposed. In order to improve the accuracy of monocular camera pose estimation, a new convolutional neural network structure is proposed. The proposed network structure adds a convolution layer and a Passthrough layer to the Darknet53 feature extraction network. The purpose is to use multi-scale feature maps in the feature extraction network for tensor splicing as the output of the feature extraction network. The feature extraction network takes a network of size 416×416×3 as input. After 16 times downsampling through the Darknet53 feature extraction network, a feature tensor of size 26×26×512 is obtained. The feature tensor that has been downsampled 16 times is passed through a convolution layer and the Passthrough layer. h layer processing, the size of the 16-fold downsampled feature tensor is modified to 13×13×2048, and the modified 16-fold downsampled feature tensor is concatenated with the 32-fold downsampled feature tensor output by the Darknet53 feature extraction network to obtain a feature tensor output of size 13×13×3072. At this time, the dimension of the feature tensor output by the feature extraction network is expanded by 3 times compared with the dimension of the feature tensor output by the traditional Darknet53, which can enable the convolutional neural network to extract local correlation features while extracting local features, making the convolutional neural network feature extraction more accurate, thereby improving the accuracy of the pose estimation algorithm. This network structure improves the accuracy of the pose estimation algorithm while ensuring the running speed of the pose estimation algorithm; The algorithm first needs to clarify the internal parameters of the camera used, and use the RGB image containing the Aruco QR code as input. The image is input into the convolutional neural network. The convolutional neural network extracts features from the 416×416 size interval based on the sliding window algorithm. After feature extraction and regression, the convolutional neural network outputs the centroid point of the target object in the RGB image in the 1-pixel coordinate system, the corner points in the 8-pixel coordinate system, and the object category prediction information; Then the algorithm recognizes the Aruco QR code in the image and establishes a three-dimensional world coordinate system based on the Aruco QR code. Since the side length of the Aruco QR code and its coordinates in the world coordinate system are known, the extrinsic parameters of the camera are solved based on the EPnP algorithm according to the pixel positions of the four corner points of the Aruco QR code in the image coordinate system. The extrinsic parameters include the rotation matrix and the translation matrix. After obtaining the rotation matrix and translation matrix of the camera when taking the current image, the centroid point of the target object in the 1-pixel coordinate system and the corner points in the 8-pixel coordinate system predicted by the convolutional neural network are transformed according to the principle of camera coordinate transformation. The coordinates of the target object in the pixel coordinate system are converted into coordinates in the world coordinate system. The position and posture of the target object are calculated according to the coordinates of the target object in the world coordinate system. Finally, the algorithm outputs the category prediction information and position and posture information of the object in the image.

Images