Three-dimensional object recognition and posture calculation method based on YOLO and depth camera

Abstract

The present invention discloses a three-dimensional object recognition and posture solution method based on YOLO and a depth camera, and relates to the field of artificial intelligence technology. The present invention includes step one: perspective transformation matrix acquisition: step two: rough positioning of objects: using laser radar to collect ground point cloud data; step three: image preprocessing: obtaining RGB images and depth images of ground objects, and changing the oblique downward perspective into a standard overlooking perspective; step four: object recognition: calling the model trained under the YOLOv5 framework for the processed RGB image to find the minimum rotation circumscribed rectangle of the object; step five: posture settlement: combining the depth camera to measure the three-dimensional coordinates of the center of the object in the real world, and converting the camera coordinates into real-world coordinates. The present invention uses laser radar to roughly locate garbage, and then uses the camera for object recognition and posture solution, so that the camera can focus on detecting objects in a small range nearby, while reducing the cost, reducing the forward distance of the camera, and improving the accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a three-dimensional object recognition and posture solving method based on YOLO and a depth camera. Background Art

[0002] In the field of 3D object recognition, 3D deep learning models, such as 3D CNN (3D convolutional neural network), PointNet, PointNet++, etc., are used to directly process 3D point cloud data or 3D mesh data to achieve object recognition and positioning, and can show good performance. However, when these models are deployed on embedded devices with limited resources, they are more difficult and cannot play their original advantages.

[0003] For example, the Chinese invention patent application with publication number CN117475228A discloses a three-dimensional point cloud classification and segmentation method based on dual-domain feature learning, which includes learning the position information according to the three-dimensional coordinates of the original input point cloud; then, incorporating the contextual position information into the channel weights of the global Transformer to strengthen the initial features of the point cloud; secondly, constructing a local graph structure for each point in the point cloud in the feature domain and the spatial domain through the K nearest neighbor algorithm, encoding the position and feature information, and using the reverse bottleneck design to update the features of the center point from the information interaction with the neighboring points; finally, stacking the dual-domain feature learning modules several times to obtain semantic information of different depths, and using these features to complete the classification and segmentation tasks of the point cloud. However, the processing of the three-dimensional point cloud in the above method is relatively redundant for simple object grasping, and the subsequent processing algorithm will occupy too many hardware resources.

[0004] Another example is the Chinese invention patent application with publication number CN115482448A, which discloses an object detection and pose estimation method based on the YOLO-6D algorithm. This method changes the YOLOv2 detection network in the original algorithm to the YOLOv3 network, and adds an attention mechanism to enhance the model's detection ability for objects with complex backgrounds and occlusions. In addition, the pose estimation method is adjusted, and a cell group is selected to perform RANSAC-based EPnP pose estimation to improve the estimation accuracy. However, the YOLOv3 network, to which the YOLOv2 detection network in the original algorithm is changed in the above method, is inferior to YOLOv5 in terms of accuracy and FPS, and the YOLO-6D algorithm cannot apply the TensorRT acceleration used by traditional YOLO, and the FPS running on Jason Nano is low. In addition, for object picking tasks, it is only necessary to determine the object angle for objects such as irregular bar-shaped objects that can only be clamped at a specific angle. If the YOLO-6D algorithm is used, it will cause a waste of resources.

[0005] Therefore, the prior art requires a more practical and efficient method for picking three-dimensional objects that is suitable for deployment on a system such as Jason Nano to reasonably utilize limited resources to complete the set tasks. Summary of the invention

[0006] The purpose of the present invention is to provide a three-dimensional object recognition and posture solution method based on YOLO and a depth camera. By using a laser radar to roughly locate the garbage, and then using a camera to perform object recognition and posture solution, the camera can be used to focus on detecting nearby objects in a small range. While reducing costs, the difficulty of applying three-dimensional object posture solution in the deployment of deep learning models is reduced, and the problem of computing power waste caused by complex algorithms in the background technology is solved.

[0007] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0008] The present invention is a three-dimensional object recognition and posture solution method based on YOLO and a depth camera, comprising the following steps:

[0009] Step 1: Perspective transformation matrix acquisition: Use the depth camera to collect long strip samples, select the four vertices of the long strip, calculate the perspective transformation matrix and store it:

[0010] Step 2: Rough object positioning: Use LiDAR to collect ground point cloud data and perform rough identification and positioning of ground objects;

[0011] Step 3: Image preprocessing: Use a depth camera to simultaneously obtain the RGB image and depth image of the ground object. Perform matrix multiplication based on the transformation matrix measured in step 1, perform inverse perspective transformation on the RGB image, and change the oblique downward perspective into a standard top-down perspective.

[0012] Step 4: Object recognition: Call the engine file to perform model inference on the processed RGB image; use the OpenCV contour search function to find the outer contour of the image based on the pre-selected box inferred by YOLOv5, and then fit the straight line in the contour based on the Hough transform to find the minimum rotated circumscribed rectangle of the object;

[0013] Step 5: Posture calculation: Based on the minimum rotation circumscribed rectangle drawn in the rotation angle detection, the three-dimensional coordinates of the center of the object in the real world are measured in combination with the depth camera, and the camera coordinates are converted into real-world coordinates to obtain the true posture of the object.

[0014] As a preferred technical solution of the present invention, it also includes brightness sample acquisition: importing training set images, and using the average grayscale value of the training set images for the reference brightness of subsequent acquired pictures to obtain the adaptability to different environments; during the image preprocessing in step three, the acquired RGB image is brightness standardized according to the acquired reference brightness to reduce the influence of ambient light.

[0015] As a preferred technical solution of the present invention, it also includes OpenCV version replacement: replacing the default configuration in the Jetson nano official image with a version that supports cuda acceleration.

[0016] As a preferred technical solution of the present invention, it also includes collecting data: calling the model trained under the YOLOv5 framework, converting the trained .pt model into a .wts file, and then deploying the .wts file to the TensorRT framework to accelerate model reasoning to improve the real-time performance of detection.

[0017] As a preferred technical solution of the present invention, in step 2, the point cloud data collected by the lidar is used to distinguish between ground and non-ground point clouds; in the non-ground point clouds, the point clouds are clustered and points with similar features are grouped into clusters to distinguish different objects.

[0018] As a preferred technical solution of the present invention, in the step 4, the position rectangle of the object is extracted as ROI (Region Of Interest). Within the ROI, the Sobel operator in OpenCV is used. Based on the discrete convolution method, the Sobel operator is applied to each pixel point of the image to obtain the gradient value of the point in the horizontal and vertical directions, thereby finding the possible edges of the image; for all contours found, the minimum bounding rectangle of each contour is found, and the largest bounding rectangle is found as the minimum rotated bounding rectangle of the object.

[0019] The present invention has the following beneficial effects:

[0020] 1. The present invention uses a laser radar to roughly locate the garbage, and then uses a camera to perform object recognition and posture calculation. The camera can be used to focus on detecting nearby objects in a small range. While reducing costs, the camera's forward-looking distance is reduced, background interference is avoided, and accuracy is improved.

[0021] 2. Based on traditional two-dimensional object recognition, the present invention uses traditional vision to solve the posture of objects, which can reduce the difficulty of applying three-dimensional object posture solution in deep learning model deployment and can greatly reduce the waste of computing power caused by complex algorithms.

[0022] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 It is a flow chart of the three-dimensional object recognition and posture solution method based on YOLO and depth camera of the present invention;

[0025] Figure 2 This is a schematic diagram of the original image collected by RealSense;

[0026] Figure 3 This is a schematic diagram of an image after inverse perspective transformation and brightness compensation;

[0027] Figure 4 This is a schematic diagram of object recognition results;

[0028] Figure 5 This is a schematic diagram of object angle detection;

[0029] Figure 6 Schematic diagram of the object's real coordinate measurement. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] With the development of industrial society, garbage sorting and picking has gradually developed towards unmanned and intelligent directions. In the prior art, garbage sorting technology based on intelligent learning technology mainly relies on computer vision to identify garbage, and then controls the mobile robot to grab the garbage and classify it. Most garbage is a three-dimensional object. In the field of three-dimensional object recognition, three-dimensional deep learning models such as 3D CNN (three-dimensional convolutional neural network), PointNet, PointNet++, etc. are used to directly process three-dimensional point cloud data or three-dimensional grid data to achieve object recognition and positioning, and can show good performance. However, when these models are deployed on embedded devices with limited resources, they are more laborious and cannot play their original advantages. Based on this, this embodiment provides a three-dimensional object recognition and posture solution method based on YOLO and a depth camera, which provides a reliable and excellent solution for the recognition and positioning of objects picked up by mobile robots, and at the same time provides a convenient and easy-to-use solution for three-dimensional object posture solution.

[0032] See also Figure 1 As shown, this embodiment provides a three-dimensional object recognition and posture solution method based on YOLO and a depth camera, which mainly includes an implementation preparation stage and an implementation stage.

[0033] The implementation preparation phase mainly includes the following steps:

[0034] Step 1: Brightness sample collection: Import the training set images, calculate the average grayscale value of the training set images, and use the average grayscale value of the training set images as the reference brightness for subsequent image collection. During the inference process, adjust the collected RGB images according to the reference brightness to obtain the adaptability to different environments. Calculate the average sample brightness based on the training set, and adjust the image average brightness to 100 to facilitate subsequent object recognition.

[0035] Step 2: Perspective transformation matrix acquisition: Use the RealSense depth camera to collect long strip samples (simulating the perspective intersection of parallel lines), select the four vertices of the long strip, calculate the perspective transformation matrix and store it for use in model reasoning.

[0036] Step 3: OpenCV version replacement: Since the default version of OpenCV in the Jetson nano official image does not support cuda acceleration, the team replaced it with a version that supports cuda acceleration to improve the efficiency of subsequent image processing.

[0037] Step 4: Collect data: Perform data enhancement operations such as random blur, rotation, and shearing on the data set to improve the accuracy of the model; then divide the data, label it, use the yolov5s model to train to get the .pt original model, and convert it into a .wts file to generate an engine file. Then deploy the .wts file to the TensorRT framework of jetsonnano to accelerate model reasoning, improve the real-time performance of detection, and export the .engine file.

[0038] The implementation phase mainly includes the following steps:

[0039] Step 5: Coarse positioning of objects: Use the point cloud data collected by the LiDAR to distinguish between ground and non-ground point clouds; in non-ground point clouds, perform cluster analysis on the point clouds, group points with similar features into clusters, and distinguish different objects. By comparing the height, shape and other features of different clusters, we can identify protrusions on the ground that may be garbage.

[0040] Step 6: Image preprocessing: Figure 2 As shown, the RealSense depth camera is turned on to obtain RGB images and depth images of ground objects at the same time. The maximum resolution of the RGB image of this depth camera is 10820P, and the resolution of the depth image is 720P. To ensure that the pixels of the depth image and the RGB image correspond one to one, the resolution is set to 720P (1280*720).

[0041] like Figure 3 As shown in the figure, firstly, the acquired RGB image is subjected to matrix multiplication according to the transformation matrix measured in step 2, thereby completing the inverse perspective transformation and changing the oblique downward perspective into a standard top-down perspective. The original image pixel position corresponding to each pixel of the inverse perspective image is saved so as to trace back the position of the target in the real world later.

[0042] At the same time, according to the sample reference brightness collected in step 1, the brightness of the acquired RGB image is normalized to reduce the influence of ambient light.

[0043] Step 7: Object recognition: Figure 4 As shown, the engine file is called for model reasoning. For objects that are affected by angles, such as bottles and batteries, the position rectangle of the object is extracted as ROI (Region Of Interest). Within the ROI, the Sobel operator in OpenCV is used. Based on the discrete convolution method, the Sobel operator is applied to each pixel point of the image to obtain the gradient value of the point in the horizontal and vertical directions, thereby finding the possible edges of the image; for all contours, the minimum bounding rectangle of each contour is found, and the largest bounding rectangle is the minimum rotated bounding rectangle of the object.

[0044] Step 8: Posture calculation: Figure 5 As shown, the real coordinates of the center of the object are obtained in the ROI in step seven, and the position of the center point of the object is traced back to the original coordinates through the original coordinate array saved during the inverse perspective transformation, so as to correspond one-to-one with the depth image.

[0045] According to the input depth map and the pixel coordinates to be converted, the depth value corresponding to the pixel in the depth map is obtained. Then, according to the camera's intrinsic parameters and depth value, the image coordinates of the pixel are converted to normalized coordinates in the camera coordinate system, and the camera coordinates are converted to real-world coordinates (such as Figure 6 At this point, the three-dimensional coordinates of the object center and the object's rotation angle are obtained, and the information can be sent to the lower computer through serial communication to control the robotic arm to grab the object.

[0046] In summary, this embodiment uses laser radar to roughly locate garbage, and then uses the camera to identify objects and perform posture calculation. The camera can focus on detecting objects in a small range nearby, which reduces the camera's forward-looking distance while reducing costs, avoiding background interference and improving accuracy. In addition, this embodiment uses traditional vision to calculate the posture of objects on the basis of traditional two-dimensional object recognition, which can reduce the difficulty of applying three-dimensional object posture calculation in deep learning model deployment and can greatly reduce the problem of computing power waste caused by complex algorithms.

[0047] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0048] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A three-dimensional object recognition and posture solution method based on YOLO and depth camera, characterized in that: The following steps are involved: Step 1: Perspective transformation matrix acquisition: Use the depth camera to collect long strip samples, select the four vertices of the long strip, calculate the perspective transformation matrix and store it: Step 2: Rough object positioning: Use LiDAR to collect ground point cloud data and perform rough identification and positioning of ground objects; Step 3: Image preprocessing: Use a depth camera to simultaneously obtain the RGB image and depth image of the ground object. Perform matrix multiplication based on the transformation matrix measured in step 1, perform inverse perspective transformation on the RGB image, and change the oblique downward perspective into a standard top-down perspective. Step 4: Object recognition: Call the engine file to perform model inference on the processed RGB image; use the OpenCV contour search function to find the outer contour of the image based on the pre-selected box inferred by YOLOv5, and then fit the straight line in the contour based on the Hough transform to find the minimum rotated circumscribed rectangle of the object; Step 5: Posture calculation: Based on the minimum rotation circumscribed rectangle drawn in the rotation angle detection, the three-dimensional coordinates of the center of the object in the real world are measured in combination with the depth camera, and the camera coordinates are converted into real-world coordinates to obtain the true posture of the object.

2. The three-dimensional object recognition and posture calculation method based on YOLO and depth camera according to claim 1, characterized in that: It also includes brightness sample collection: importing training set images, and using the average grayscale value of the training set images as the reference brightness for subsequent collection of images to obtain the adaptability to different environments; During the image preprocessing in step three, the brightness of the acquired RGB image is normalized according to the collected reference brightness to reduce the influence of ambient light.

3. The three-dimensional object recognition and posture solving method based on YOLO and depth camera according to claim 1, characterized in that: It also includes OpenCV version replacement: replace the default configuration in the Jetson nano official image with a version that supports CUDA acceleration.

4. The three-dimensional object recognition and posture solving method based on YOLO and depth camera according to claim 1, characterized in that: It also includes data collection: calling the model trained under the YOLOv5 framework, converting the trained .pt model into a .wts file, and then deploying the .wts file to the TensorRT framework to accelerate model reasoning.

5. The three-dimensional object recognition and posture solving method based on YOLO and depth camera according to claim 1, characterized in that: In the step 2, the point cloud data collected by the laser radar is used to distinguish between ground and non-ground point clouds; in the non-ground point clouds, the point clouds are clustered and points with similar features are grouped into clusters to distinguish different objects.

6. The three-dimensional object recognition and posture solving method based on YOLO and depth camera according to claim 1, characterized in that: In the step 4, the position rectangle of the object is extracted as ROI (Region Of Interest). In the ROI, the Sobel operator in OpenCV is used based on discrete convolution to apply the Sobel operator to each pixel of the image to obtain the gradient value of the point in the horizontal and vertical directions, thereby finding the possible edge of the image; For all the contours found, find the minimum enclosing rectangle of each contour, and the largest enclosing rectangle is the minimum rotated enclosing rectangle of the object.

Images