Mechanical arm visual guidance point cloud scanning method based on deep learning

Through deep learning and improved YOLOv5s object detection model, combined with robotic arm and binocular camera, automatic high-precision three-dimensional measurement of car body clearance is achieved, solving the problems of slow manual measurement speed and low accuracy, and meeting the needs of automated production and assembly .

CN118952193APending Publication Date: 2024-11-15NANJING YUNTONG TECH CO LTD

Patent Information

Application Number
CN202411065003.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing automotive body clearance surface difference measurement tools rely on manual operation, and are slow and have low accuracy, making it difficult to meet the needs of automated production and assembly.

Method used

Using the deep learning-based robotic arm visually guided point cloud scanning method, images and depth data are collected through binocular cameras, and automatic recognition of target points and three-dimensional coordinate conversion is achieved using the improved YOLOv5s target detection model, and the robotic arm is driven to perform point cloud scanning. Obtain high-precision three-dimensional spatial measurements.

Benefits of technology

It realizes automatic, accurate and efficient three-dimensional spatial measurement of car clearance, improves detection speed and accuracy, and solves the problems of cumbersome and inefficient manual operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118952193A_ABST
    Figure CN118952193A_ABST
Patent Text Reader

Abstract

The invention discloses a mechanical arm visual guidance point cloud scanning method based on deep learning. The method comprises the steps that a circular punctuation point is pasted to the upper right corner of a to-be-detected gap; a binocular camera on a tail end shaft of the mechanical arm collects a color image and a depth image and carries out registration; detecting a target point in real time; querying the registered depth image through a two-dimensional coordinate of a target point center in a pixel coordinate system obtained by a target detection network, and obtaining a depth value corresponding to the target point center in a camera coordinate system; the center coordinate of the target point is converted into a world coordinate system from a pixel coordinate system and sent to a mechanical arm; and after receiving the coordinates of the target point, the mechanical arm drives the point cloud scanning equipment at the tail end to move to the target point, scanning action is executed, and the point cloud scanning equipment transmits scanned assembling gap point cloud data to the upper computer. According to the invention, the gap of each part of the automobile can be rapidly positioned and accurately scanned in a complex scene, and the problems of tedious manual operation, low efficiency and limited measurement precision in a traditional method are effectively solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of vision guidance based on deep learning, and specifically relates to a robot arm vision-guided point cloud scanning method based on deep learning. Background Art

[0002] During the automobile assembly process, due to subsequent assembly or product appearance requirements, it is necessary to detect and control the gaps and flushness of multiple parts of the car body. At present, the gap and flushness measurement tools for automobile bodies mainly include gap gauges and flushness gauges, and the measurement method is for the measurement personnel to measure with handheld tools. However, the manual measurement method has a slow detection speed, and the scanning speed of the gap area affects the measurement accuracy. It is difficult to meet the needs of automated production and assembly.

[0003] With the rapid development of deep learning, it has been widely used in the field of industrial production. Deep learning simulates the learning process of the human brain by building artificial neural networks, and realizes efficient processing and analysis of complex data. In the field of industrial production, deep learning is widely used in quality inspection, predictive maintenance, intelligent manufacturing, etc., greatly improving production efficiency and product quality. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a deep learning-based robotic arm vision-guided point cloud scanning method, which can realize automatic, accurate and efficient three-dimensional spatial measurement of automobile gaps.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] The method for point cloud scanning based on deep learning and guided by vision of a robotic arm includes the following steps:

[0007] S1. Set the car model to be tested and the gap type to be scanned, load the corresponding robot arm scanning action, and paste a circular punctuation mark in the upper right corner of the gap to be tested;

[0008] S2. After the vehicle body to be inspected is sent into the inspection system by the conveying mechanism and fixed, the industrial robot arm starts the automatic point-finding mode, and the binocular camera on the end axis of the robot arm starts to collect color images and depth images, and performs D2C registration;

[0009] S3, the host computer's deep learning-based target detection network detects real-time target points in the color image collected by the binocular camera. When the target point is detected, a signal is sent to the robotic arm to stop the point-finding movement;

[0010] S4, after the robot arm stops searching for points, the two-dimensional coordinates of the center of the target point in the pixel coordinate system are obtained through the target detection network, and the depth image after D2C registration is queried to obtain the depth value of the center of the target point in the camera coordinate system;

[0011] S5. According to the intrinsic parameters of the binocular camera, the hand-eye calibration matrix from the end of the robotic arm to the binocular camera, and the depth value, a rigid body transformation is applied to convert the center coordinates of the target point from the pixel coordinate system to the three-dimensional coordinates in the world coordinate system and send them to the robotic arm;

[0012] S6. After receiving the three-dimensional coordinates of the target point in the world coordinate system, the robot arm drives the point cloud scanning device at the end to move to the target point and starts scanning. The point cloud scanning device transmits the scanned assembly gap point cloud data to the host computer.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] In the above step S1, the circular punctuation mark adopts a structure in which a small circle in the center is nested with a circular ring, and the colors from the inside to the outside are white and black respectively.

[0015] The binocular camera in the above step S2 includes one RGB color camera and two infrared cameras.

[0016] In the above step S2, the automatic point-finding mode of the robot arm is to make a spiral motion that gradually spreads outward with the current end point as the center, and its motion trajectory equation is:

[0017]

[0018] Where b is the increase of p for every 1 degree rotation; t is time; p is the rotation radius; x, y, z are the coordinate points in the world coordinate system of the robot.

[0019] In the above step S3, the target detection network based on deep learning is a target detection model improved based on YOlOV5s. During training, a vehicle body punctuation image dataset is required. The dataset is divided into a training set, a test set, and a validation set according to a ratio of 7:2:1.

[0020] Pre-training is performed on the dataset, and the pre-training weights are loaded through transfer learning to obtain the optimal deep learning-based object detection network.

[0021] The above-mentioned target detection model improved based on YOlOV5s adds auxiliary reversible branches during training to help the backbone feature extraction network to better update parameters during the training process. The backbone feature extraction network uses MobileNetV3 for feature extraction.

[0022] The above-mentioned target detection model based on the improvement of YOLOv5s uses the GConv module to replace the Conv convolution module in YOLOv5s. The GConv module first performs 1x 1 convolution to aggregate the information features between channels, and then uses grouped convolution to divide the feature channels into 2 groups, which are convolved separately and then spliced ​​as the output of the GConv module.

[0023] The above-mentioned target detection model based on the improvement of YOLOv5s uses the C3G module to replace the BottleneckC3 in YOLOv5s, retaining the feature reuse characteristics of the C3 module, that is, adding the features of the previous stage to the features processed in the current stage to form a cross-stage feature fusion, which is conducive to gradient flow and feature reuse. The introduction of the GConv structure outputs more feature channels without significantly increasing the amount of calculation, thereby realizing the expansion of feature dimensions.

[0024] In the above step S3, the target detection network based on deep learning detects the punctuation points in the image, obtains the vertex coordinates of the minimum circumscribed rectangle of the punctuation points and the confidence of the punctuation points, and takes the punctuation points with confidence higher than the confidence threshold as the detected target points.

[0025] In the above step S5, the steps of converting the target point center coordinates from the pixel coordinate system to the three-dimensional coordinates in the world coordinate system are:

[0026] First, convert the center coordinates of the target point from the pixel coordinate system to the camera coordinate system. The conversion formula is:

[0027]

[0028] Where x cam ,y cam is the center coordinate of the target point, k0, k1...k8 are the camera internal parameters, z cam It is the depth value of the target point center in the camera coordinate system;

[0029] Then convert the coordinates from the camera coordinate system to the coordinates in the world coordinate system. The formula is:

[0030]

[0031] Where x world ,y world , z world is the world coordinate system of the target point center, D1 is the hand-eye calibration matrix, and D2 is the transformation matrix of the current end coordinate system of the robot arm relative to the world coordinate system.

[0032] The method for obtaining the above hand-eye calibration matrix D1 is:

[0033] The robotic arm drives the end color camera to take pictures of the calibration plate at different points according to a set of preset calibration actions, and at least two joints are guaranteed to have relative motion between every two points;

[0034] By identifying the features of the calibration plate, the transformation matrix from pixel coordinates to the calibration plate coordinate system is obtained. Query the transformation matrix of the end of the robot arm relative to the base coordinate system at this time By combining multiple sets of calibration plates to the base coordinate system transformation matrix Solve the hand-eye calibration matrix from the end of the robot arm to the camera coordinate system. The equation is:

[0035]

[0036] The calibration plate coordinate system is as follows: the upper left corner of the calibration plate is taken as the origin, the direction perpendicular to the calibration plate is the Z axis direction, and the length and width directions of the calibration plate are the X and Y axis directions to establish a coordinate system.

[0037] In the above step S6, after receiving the three-dimensional coordinates of the target point in the world coordinate system, the robotic arm calculates the direction of the line connecting the target point and the current end, and uses the world coordinates 30 cm away from the mark as the position of the planned point of the robotic arm end. The posture of the robotic arm end at the planned point ensures that the Z axis of the end is facing the mark, and the Y axis direction ensures that the camera is level.

[0038] The present invention has the following beneficial effects:

[0039] The target detection network of the present invention uses a target detection model based on the improvement of YOLOv5s. The GConv structure replaces the Conv convolution module in YOLOv5s, effectively increasing the width of the network (i.e., the number of channels) without significantly increasing the computational cost, outputting more feature channels, and realizing feature dimension expansion; the C3G module is used to replace the BottleneckC3 in YOLOv5s, retaining the feature reuse characteristics of the C3 module, that is, adding the features of the previous stage to the features processed in the current stage, forming a cross-stage feature connection design, promoting feature reuse and information flow within the model, facilitating gradient propagation, accelerating the training process, and improving the stability of the model. Accurate recognition and positioning of punctuation points is realized

[0040] The present invention is based on a binocular vision system, an industrial robot arm, and a point cloud scanning system, and uses deep learning to achieve robot arm vision-guided point cloud scanning, which mainly includes the following parts: in actual application, the binocular camera is fixed to the end of the industrial robot arm through a clamp, and the worker pastes a circular punctuation mark on the upper right of the automobile body assembly gap to be detected, and the RGB camera carried on the binocular camera system obtains the color image of the front car, and the depth camera obtains the depth image, and the visual target detection model based on deep learning is used to detect the punctuation marks in the image, and the two-dimensional coordinates of the punctuation marks obtained by the detection are queried after completing D2C registration with the depth image to obtain its depth value, and the coordinate conversion is applied to convert it into a target point in the 3D world coordinate system, and the target point is sent to the industrial robot arm, and the robot arm performs a scanning action, and the point cloud scanning system scans the area containing the body assembly gap to be detected to obtain high-precision point cloud data. The present invention can quickly locate and accurately scan the gaps of various parts of the car in complex scenes, effectively solving the problems of cumbersome manual operation, low efficiency and limited measurement accuracy in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the robot arm vision-guided point cloud scanning method of the present invention.

[0042] Figure 2 A 6-axis industrial robot arm used in the present invention.

[0043] Figure 3 It is the punctuation point of the present invention and the gap to be detected.

[0044] Figure 4 This is a diagram of the improved module structure of YOLOv5s in the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0047] like Figure 1-3As shown, the robot arm vision-guided point cloud scanning method based on deep learning of the present invention is implemented by using a binocular vision system, an industrial robot arm, and a point cloud scanning system. Its workflow is as follows: a worker pastes a circular punctuation mark on the upper right corner of the assembly gap of the car body that needs to be detected, and obtains a color image of the car in front through the RGB camera carried by the binocular camera system, and the depth camera obtains a depth image. The punctuation marks in the image are detected using a visual target detection model based on deep learning. The two-dimensional coordinates of the punctuation marks obtained by the detection are queried for their depth values ​​after completing D2C registration with the depth image. The coordinate transformation is applied to convert them into target points in the three-dimensional world coordinate system, and the target points are sent to the industrial robot arm. The robot arm performs a scanning action, and the point cloud scanning system scans the area containing the assembly gap of the car body to be detected to obtain high-precision point cloud data. The system can quickly locate and accurately scan the gaps between various parts of the car body in complex scenes, effectively solving the problems of cumbersome manual operation, low efficiency and limited measurement accuracy in traditional methods. Specifically, the technical solution steps taken to implement the present invention are as follows:

[0048] S1. Before the assembled automobile body enters the inspection system, workers are required to paste a circular mark on the upper right corner of the inspection gap.

[0049] In the embodiment, before the assembled car body enters the detection system, the operator sets the car model to be tested and the type of gap to be scanned (such as door and body, front and rear bumpers, etc.) through the human-computer interaction interface. The system loads the corresponding robotic arm scanning action and pastes a circular punctuation mark in the upper right corner of the gap to be inspected.

[0050] The structure of the circular punctuation mark is composed of a small circle in the center and a ring, and the colors from the inside to the outside are white and black respectively.

[0051] S2. After the vehicle body to be inspected is sent into the inspection system by the conveying mechanism and fixed, the industrial robot arm starts the automatic point-finding mode. The binocular camera fixed on the end axis of the robot arm by the clamp starts to collect color images and depth images, and performs D2C registration.

[0052] In the embodiment, the binocular camera consists of 1 RGB color camera and 2 infrared cameras.

[0053] The automatic point-finding mode of the robot arm is to make a spiral motion that gradually spreads outward with the current end point as the center. The motion trajectory equation is:

[0054]

[0055] Where b is the increase of p for every 1 degree rotation. t is the time. p is the rotation radius. x, y, z are the coordinate points in the world coordinate system of the robot.

[0056] S3, the color image collected by the binocular camera is transmitted to the host computer through the network port, and the target detection model deployed on the host computer detects the collected image in real time. When the target point is detected, a signal is sent to the robot arm to stop the point-finding movement.

[0057] In the embodiment, when training a deep learning-based object detection network, a vehicle body punctuation image dataset is required, and the dataset is divided into a training set, a test set, and a validation set at a ratio of 7:2:1. By pre-training on a general dataset and loading the pre-training weights through transfer learning, the number of dataset samples required for the training model is greatly reduced, reducing the construction cost.

[0058] The target detection network based on deep learning is a target detection network improved based on YOLOv5s. An auxiliary reversible branch is added in the training stage to help the backbone feature extraction network to better update parameters during the training process. The branch is discarded in the reasoning stage to speed up the reasoning speed.

[0059] like Figure 4 As shown in the figure, the target detection model improved based on YOLOv5s uses the GConv module to replace the Conv convolution module in YOLOv5s. The GConv module first performs 1x 1 convolution to aggregate information features between channels, and then uses grouped convolution to divide the feature channels into 2 groups, which are convolved separately and then concatenated as the output of the GConv module.

[0060] The target detection model improved based on YOLOv5s uses the C3G module to replace BottleneckC3 in YOLOv5s, retaining the feature reuse characteristics of the C3 module, that is, adding the features of the previous stage to the features processed in the current stage to form a cross-stage feature fusion.

[0061] The punctuation points in the image are detected by the deep learning network, and the vertex coordinates of the minimum circumscribed rectangle of the circular punctuation points and the confidence of the punctuation points are obtained. By setting the confidence threshold, the punctuation points detected by the detection model with a confidence level lower than 0.5 are removed.

[0062] S4. When the robot arm stops searching for points, the two-dimensional coordinates of the center of the mark in the pixel coordinate system are obtained by detecting the model, and the depth map after registration is queried to obtain the depth value of the center of the mark from the camera coordinate system.

[0063] S5. Obtain the coordinate transformation matrix from the pixel coordinate system to the world coordinate system based on the binocular camera internal parameters and the hand-eye calibration matrix of the robotic arm and the binocular camera. Use the obtained depth value to apply the rigid body transformation to convert the 2D coordinates of the punctuation point center into 3D coordinates in the world coordinate system and send them to the robotic arm.

[0064] In the embodiment, the conversion step of the punctuation point center coordinates from the pixel coordinate system to the world coordinate system is to first convert from the pixel coordinate system to the camera coordinate system, and the conversion formula is:

[0065]

[0066] Where x cam ,y cam is the pixel coordinate of the center point, k0, k1...k8 are the camera internal parameters, z cam It is the depth value corresponding to the center point in the camera coordinate system.

[0067] Then convert from camera coordinates to world coordinates. The formula is:

[0068]

[0069] Where x world ,y world , z world is the world coordinate system of the center point, D1 is the calibrated hand-eye calibration matrix, and D2 is the transformation matrix of the current end coordinate system of the robot arm relative to the world coordinate system.

[0070] The method to obtain the D1 hand-eye calibration matrix is:

[0071] The robotic arm drives the end color camera to take pictures of the calibration plate at different points according to a set of preset calibration actions, and at least two joints are guaranteed to have relative movement between every two points.

[0072] By identifying the features of the calibration plate, the transformation matrix from pixel coordinates to the calibration plate is obtained Query the transformation matrix of the end of the robot arm relative to the base coordinate system at this time By combining multiple sets of conversion equations from calibration plates to the end of the camera, the hand-eye calibration matrix from the end of the robot arm to the camera coordinate system is solved. The conversion equation is:

[0073]

[0074] Where: is the transformation matrix of the end of the robot relative to the base coordinate system, The hand-eye calibration matrix D1 from the end of the robot arm to the camera coordinate system needs to be solved. is the coordinate transformation matrix from camera to calibration plate, is the transformation matrix from the calibration plate to the camera base coordinate system, which is a fixed value.

[0075] The calibration plate coordinate system is established by taking the upper left corner of the calibration plate as the origin, the direction perpendicular to the calibration plate as the Z axis, and the length and width directions of the calibration plate as the X and Y axis directions.

[0076] S6. After receiving the target point, the robotic arm drives the point cloud scanning device at the end to move to a distance of 30 cm from the target point and starts scanning. The point cloud scanning device transmits the scanned assembly gap point cloud data to the host computer.

[0077] In the embodiment, after receiving the world coordinates of the punctuation point, the robotic arm calculates the direction of the line connecting the distance punctuation point and the current end, and uses the world coordinates 30 cm away from the punctuation point as the position of the planned point of the robotic arm end. The posture of the robotic arm end at the planned point ensures that the Z axis of the end is facing the punctuation point, and the Y axis direction ensures that the camera is level.

[0078] The point cloud scanning device is a high-resolution laser radar or structured light scanning device mounted on the end of the robotic arm, which can generate three-dimensional point cloud data of the surface and gap area of ​​the vehicle to be tested in real time.

[0079] After the host computer receives the point cloud data, it uses the point cloud processing algorithm to perform preprocessing, including removing noise points, filtering and smoothing, to improve the quality of the point cloud data.

[0080] Next, the point cloud registration algorithm is used to fuse the currently scanned point cloud data with the historical scan data to construct a complete three-dimensional model of the vehicle body assembly gap.

[0081] By performing geometric analysis on the fused point cloud data, such as edge detection and surface fitting, the size and shape of the assembly gap are calculated and the assembly quality is evaluated.

[0082] Based on the evaluation results, the system automatically generates a report including the test results, deviation analysis and possible problems. The report can be displayed in real time on the operator interface for personnel to view.

[0083] If the test results meet the preset standards, the system will continue to wait for the next car to be inspected; if not, the system will trigger an alarm mechanism to prompt workers to make adjustments or repairs.

[0084] During the entire workflow, the system monitors the status of the robotic arm and point cloud scanning equipment in real time to ensure the normal operation of the equipment, and records all operation logs for subsequent data analysis and system optimization.

[0085] The above steps constitute a complete workflow of the deep learning-based robotic arm vision-guided point cloud scanning method, which realizes automated and high-precision automobile body assembly gap detection and improves production efficiency and detection accuracy.

[0086] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0087] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A robotic arm vision-guided point cloud scanning method based on deep learning, It is characterized in that The following steps are involved: S1. Set the car model to be tested and the gap type to be scanned, load the corresponding robot arm scanning action, and paste a circular punctuation mark in the upper right corner of the gap to be tested; S2. After the vehicle body to be inspected is sent into the inspection system by the conveying mechanism and fixed, the industrial robot arm starts the automatic point-finding mode, and the binocular camera on the end axis of the robot arm starts to collect color images and depth images, and performs D2C registration; S3, the host computer's deep learning-based target detection network detects real-time target points in the color image collected by the binocular camera. When the target point is detected, a signal is sent to the robotic arm to stop the point-finding movement; S4, after the robot arm stops searching for points, the two-dimensional coordinates of the center of the target point in the pixel coordinate system are obtained through the target detection network, and the depth image after D2C registration is queried to obtain the depth value of the center of the target point in the camera coordinate system; S5. According to the intrinsic parameters of the binocular camera, the hand-eye calibration matrix from the end of the robotic arm to the binocular camera, and the depth value, a rigid body transformation is applied to convert the center coordinates of the target point from the pixel coordinate system to the three-dimensional coordinates in the world coordinate system and send them to the robotic arm; S6. After receiving the three-dimensional coordinates of the target point in the world coordinate system, the robot arm drives the point cloud scanning device at the end to move to the target point and starts scanning. The point cloud scanning device transmits the scanned assembly gap point cloud data to the host computer.

2. According to the deep learning-based robotic arm vision-guided point cloud scanning method of claim 1, It is characterized in that In step S1, the circular punctuation mark adopts a structure in which a small circle in the center is connected to a circular ring, and the colors from the inside to the outside are white and black respectively.

3. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that The binocular camera in step S2 includes one RGB color camera and two infrared cameras.

4. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that In step S2, the automatic point-finding mode of the robot arm is to make a spiral motion that gradually spreads outward with the current end point as the center, and its motion trajectory equation is: Where b is the increase of p for every 1 degree rotation; p is the rotation radius; t is the time; x, y, z are the coordinate points in the world coordinate system of the robot.

5. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that In step S3, the target detection network based on deep learning is a target detection model improved based on YOLOv5s. During training, a vehicle body punctuation image dataset is required. The dataset is divided into a training set, a test set, and a validation set according to a ratio of 7:2:

1. Pre-training is performed on the dataset, and the pre-training weights are loaded through transfer learning to obtain the optimal deep learning-based object detection network.

6. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 5, It is characterized in that The improved target detection model based on YOLOv5s adds auxiliary reversible branches during training to help the backbone feature extraction network to better update parameters during training. The backbone feature extraction network uses MobileNetV3 for feature extraction. The Conv convolution module in YOLOv5s is replaced by the GConv module. The GConv module first performs 1x 1 convolution to aggregate information features between channels, and then uses grouped convolution to divide the feature channels into 2 groups, which are convolved separately and then spliced ​​as the output of the GConv module. The C3G module is used to replace the BottleneckC3 in YOLOv5s.

7. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that In step S3, the target detection network based on deep learning detects the punctuation points in the image, obtains the vertex coordinates of the minimum circumscribed rectangle of the punctuation points and the confidence of the punctuation points, and takes the punctuation points with confidence higher than the confidence threshold as the detected target points.

8. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that In step S5, the steps of converting the target point center coordinates from the pixel coordinate system to the three-dimensional coordinates in the world coordinate system are: First, convert the center coordinates of the target point from the pixel coordinate system to the camera coordinate system. The conversion formula is: Where x cam ,y cam is the center coordinate of the target point, k 0 , k 1 ...k 8 is the camera internal parameter, z cam It is the depth value of the target point center in the camera coordinate system; Then convert the coordinates from the camera coordinate system to the coordinates in the world coordinate system. The formula is: Where x world ,y world , z world is the world coordinate system of the target point center, D 1 is the hand-eye calibration matrix, D 2 It is the transformation matrix of the current end coordinate system of the robot relative to the world coordinate system.

9. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 8, It is characterized in that The hand-eye calibration matrix D 1 The method to obtain is: The robotic arm drives the end color camera to take pictures of the calibration plate at different points according to a set of preset calibration actions, and at least two joints are guaranteed to have relative motion between every two points; By identifying the features of the calibration plate, the transformation matrix from pixel coordinates to the calibration plate coordinate system is obtained. Query the transformation matrix of the end of the robot arm relative to the base coordinate system at this time By combining multiple sets of calibration plates to the base coordinate system transformation matrix Solve the hand-eye calibration matrix from the end of the robot arm to the camera coordinate system. The equation is: The calibration plate coordinate system is as follows: the upper left corner of the calibration plate is taken as the origin, the direction perpendicular to the calibration plate is the Z axis direction, and the length and width directions of the calibration plate are the X and Y axis directions to establish a coordinate system.

10. The method for point cloud scanning based on deep learning and guided by vision of a robotic arm according to claim 1, It is characterized in that In step S6, after receiving the three-dimensional coordinates of the target point in the world coordinate system, the robotic arm calculates the direction of the line connecting the target point and the current end, and uses the world coordinates 30 cm away from the mark as the position of the planned point of the robotic arm end. The posture of the robotic arm end at the planned point ensures that the Z axis of the end is facing the mark, and the Y axis direction ensures that the camera is level.

Citation Information

Patent Citations

  • Bending detection method for bar steel material

    CN112950545A

  • Method and device for detecting vehicle door gap parameters, and measuring equipment

    CN113850786A

  • Image processing system and method, intelligent terminal and computer readable storage medium

    CN114862658A

  • Data processing method and device and storage medium

    CN115330917A

  • Blind search method for spaceflight shaft hole assembly right triangle hypotenuse

    CN117984326A

Cited By

  • Vehicle door assembly guiding method and system based on high-precision positioning 3D camera

    CN119600067A

  • Visual servo-based in-vivo restoration system and restoration control method

    CN119773236A

  • Visual positioning method for UHPC highway guardrail center hole

    CN120726126A