Device sorting system based on image recognition technology

Abstract

The invention belongs to the technical field of automation, and discloses a device sorting system based on an image recognition technology, which comprises a mechanical arm, the mechanical arm is a multi-degree-of-freedom mechanical arm, a mechanical claw and a camera are arranged on the mechanical arm, and the device sorting system further comprises a server connected with the mechanical arm, the mechanical claw and the camera, an image processing and recognizing module, a coordinate conversion module, a mechanical arm motion planning module, a mechanical arm control module, a mechanical claw control module, a data recording and analyzing module and a user interaction interface are arranged in the server; the method is high in automation degree and high in recognition precision, and effectively improves the device processing efficiency.

Description

Technical Field

[0001] The invention belongs to the field of automation technology and relates to a classification system, in particular to a device sorting system based on image recognition technology. Background Art

[0002] In recent years, the warehousing and logistics industry has developed rapidly. With the increase in output and the improvement in logistics speed, the efficiency of manual sorting has gradually been unable to meet the industry's urgent demand for high-efficiency and high-accuracy sorting systems; the traditional manual device sorting method is inefficient and highly dependent on manpower in harsh working environments; the automated device sorting system can significantly improve the speed and accuracy of device sorting, reduce labor costs and improve processing capabilities; at the same time, with the rapid development of computer vision and deep learning technology, it provides a technical foundation for the application of image recognition technology in the field of device sorting. Using these advanced technologies, devices can be identified and classified more accurately, improving the intelligence level of the system.

[0003] With the increase in production volume and the improvement in logistics speed, traditional manual sorting methods can no longer meet the needs of modern society, especially in the context of increasingly complex device specifications; the device sorting working environment is very complex, requiring high proficiency of staff, and manual sorting is greatly affected by environmental factors such as light and temperature; manual sorting is prone to errors, especially when the device specifications are numerous and the appearance is similar, accurate classification is very difficult.

[0004] Therefore, the existing technology has limitations in the accuracy of device sorting, processing speed and adaptability to complex environments; and the existing automated sorting technology has not yet been popularized or has problems with insufficient efficiency and accuracy, and needs to be updated in combination with the latest image recognition and mechanical control technologies. Summary of the invention

[0005] The object of the present invention is to provide a device sorting system based on image recognition technology which has high automation, high recognition accuracy and effectively improves device processing efficiency.

[0006] Based on the above purpose, the present invention adopts the following technical solution: a device sorting system based on image recognition technology, including a robotic arm, which is a multi-degree-of-freedom robotic arm. A robotic claw and a camera are provided on the robotic arm. It also includes a server connected to the robotic arm, the robotic claw and the camera. The server is provided with an image processing and recognition module, a coordinate conversion module, a robotic arm motion planning module, a robotic arm control module, a robotic claw control module, a data recording and analysis module and a user interaction interface.

[0007] The following steps are used to sort the devices:

[0008] Step 1: The camera captures the device image and transmits it to the server;

[0009] Step 2: The image processing and recognition module in the server processes the collected device image to identify the device type and location;

[0010] Step 3: The coordinate conversion module converts the recognition result obtained in step 2 from image coordinates to actual space coordinates;

[0011] Step 4: The robot arm motion planning module calculates the joint angles and motion paths of the robot arm based on the actual spatial coordinates obtained in step 3;

[0012] Step 5: The robot arm control module controls the robot arm to move above the device according to the calculation result of step 4; the robot claw control module receives the signal and performs the grasping action;

[0013] Step 6: The robot arm moves the device to the designated classification area and releases it with the robot claw;

[0014] Step 7: The user interaction interface displays the sorting process and system status in real time and receives user instructions;

[0015] Step 8: The data recording and analysis module records the sorting results and provides a basis for performance optimization.

[0016] In step 2, the image processing and recognition module uses the YOLOv5 deep learning algorithm. YOLOv5 will use a TXT format data set, which includes the types of items contained in the training data set image, the center point coordinates of the item bounding box, and the width and height of the bounding box. These five data are formed into a .txt suffix file in the prescribed format, which is the training data set. This patent first divides the target devices to be sorted into three categories: similar to spherical tools, conical tools, and cylindrical tools. The workpieces are first roughly classified into spheres, cylinders, and cones according to the geometric features of the target, and then subdivided into categories of different specifications according to the local size features of the object. Through step-by-step labeling training, the complexity of labeling is reduced and the labeling efficiency is improved. By dividing the detection tasks into two different levels of coarse and fine, the detection accuracy of a single detection task is improved; and the fourth category of other devices is not included in the recognition category of this experiment, which can reduce the error rate of identifying the background as a target and avoid unnecessary trouble in engineering. According to the device category, label the data and make a data set. Use the labeling tool Labelimg to import the crawled images and the actual collected images and manually label them to form a data set of 600 images. Put the images and annotation files in different folders and name them images and labels respectively. At the same time, you can also create two folders, train and val, in the images folder, which represent the data sets used to train the model and test the model, namely the training set and the validation set.

[0017] Set network training parameters such as learning rate, and use the data set to train YOLOv5 to obtain a detection model. Use the detection model to identify devices, and output the category, location / pixel coordinates, and size information of the device in the image for the input image.

[0018] In step 3, the coordinate conversion module uses the Zhang Zhengyou calibration method to calibrate the camera's internal and external parameters. In order to determine the position of the object to be grasped in the real world (world coordinate system) through the information captured by the camera (pixel coordinate system), the coordinate conversion module is required to realize the conversion between coordinate systems. The relevant coordinate systems are introduced as follows: World coordinate system (X W , Y W , Z W ) is also called the reference coordinate system. The coordinate system established with the center point of the robot base as the origin is the base coordinate system, which is generally used as the world coordinate system. C , Y C , Z C ) is a coordinate system established with the optical center of the camera as the origin. The pixel coordinate system (u, v) is a coordinate system established with the upper left corner of the image as the origin and pixel as the unit length. The image coordinate system (x, y) is established to facilitate the conversion from the camera coordinate system to the pixel coordinate system, and introduces the projection transmission relationship of the object from the camera coordinate system to the image coordinate system.

[0019] There is a relationship between the image coordinate system (x, y) and the pixel coordinate system (u, v). 0 , v 0 ) offset, and according to the size conversion relationship between the image coordinate system and the pixel coordinate system, the x or y axis size of each pixel in the image coordinate system is dx or dy. In summary, the following equation can be derived from the geometric relationship:

[0020]

[0021] Based on the pinhole imaging model, the image coordinate system (x, y) and the camera coordinate system (X C , Y C , Z C )relation:

[0022]

[0023] Where f is the focal length. The relationship between the pixel coordinate system and the camera coordinate system can be further calculated:

[0024]

[0025] definition Then f x 、f y 、u 0 、v0 The checkerboard is used as a calibration tool, and the camera intrinsic parameters are calibrated using the Matlab calibration toolbox.

[0026] The relationship between the camera coordinate system and the world coordinate system can be described as:

[0027]

[0028] Among them, R is the rotation matrix, t is the translation matrix, is the camera extrinsic matrix, which is calculated by Zhang Zhengyou calibration method. Combined with the extrinsic parameter matrix, the pixel coordinates can be transformed into the world coordinate system.

[0029] In step 4, the robot arm motion planning module uses inverse kinematics calculation. The posture of the robot arm end effector is expressed by Euler angles, and the calculated target posture is expressed by rotation matrix. First, the target posture is converted into Euler angles through the conversion relationship between Euler angles and rotation matrix, that is, the target posture of the robot arm, and path planning is performed based on the robot arm API.

[0030] Compared with the prior art, the present invention realizes automatic identification, classification and sorting of devices by combining advanced image recognition algorithms, robotic arm control technology and automation system integration;

[0031] The present invention is based on the application of deep learning image recognition technology, and adopts the YOLOv5 model for device image recognition and classification. The high speed and accuracy have significant advantages in the field of target detection, and can effectively handle the real-time recognition requirements in device sorting.

[0032] Aiming at the problem of device classification, the present invention adopts crawled pictures and actual collected pictures to produce device classification data sets, thereby ensuring the quality of training data and enhancing the adaptability and practicality of the system;

[0033] The present invention uses Zhang Zhengyou calibration method to achieve high-precision hand-eye calibration between the camera and the robot arm, ensuring the accurate position determination of the device in three-dimensional space, and providing a basis for the precise grasping of the robot arm;

[0034] The camera of the present invention uses a structured light 3D camera to obtain the depth information of the device, which can achieve accurate measurement of the three-dimensional shape and position of the device, especially when it can effectively process irregularly shaped or overlapping devices;

[0035] The present invention uses inverse kinematics calculation to accurately calculate the required joint angle according to the posture of the end effector of the robot arm and the position of the target object, thereby achieving precise control and grasping of the robot arm;

[0036] The robot arm of the present invention adopts a six-degree-of-freedom robot arm and an electric mechanical claw. Based on the API interface of the robot arm control, the target position is sent through network communication, and the robot arm posture is obtained in real time. For the electric mechanical claw, serial communication is used to send instructions and obtain the current state. Based on multi-threaded programming, the control of the robot arm posture and the opening and closing of the mechanical claw is realized to complete complex device sorting tasks, realizing the flexibility and efficiency of the grasping system;

[0037] The system design of the present invention adopts modularity. From image recognition, coordinate conversion to mechanical control, each part can be independently optimized and upgraded, which is convenient for future maintenance and functional expansion. It can be expanded to more device types and more complex sorting scenarios, and has good adaptability and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a work flow chart of the present invention;

[0039] Figure 2 It is a schematic diagram of device grabbing in the present invention;

[0040] Figure 3 This is an example diagram of a target device to be sorted in the present invention;

[0041] Figure 4 This is the recognition intermediate effect diagram of the deep learning algorithm in the present invention;

[0042] Figure 5 This is the final effect diagram identified in the present invention. DETAILED DESCRIPTION

[0043] Device sorting systems based on image recognition technology, such as Figure 1-5 As shown, it includes a robotic arm, which is a multi-degree-of-freedom robotic arm. A six-degree-of-freedom collaborative robotic arm can be used to grasp and carry devices. According to the load capacity, working range and accuracy requirements, industrial robotic arms of different brands (such as KUKA, Fanuc, ABB, etc.) and models can be selected. The robotic arm is provided with a mechanical claw and a camera; the camera adopts a depth camera, and the 2D image and 3D position information of the device are obtained through the depth camera; the mechanical claw is connected to the end of the robotic arm to grasp the device, and mechanical claws of different shapes (such as two fingers, three fingers or multiple fingers) and sizes can be designed according to the size and shape of the device. At the same time, a modular design can be adopted, and the components of the mechanical claw can be quickly replaced or upgraded according to different tasks; the device sorting system also includes a server connected to the robotic arm, the mechanical claw and the camera, and the server is provided with an image processing and recognition module, a coordinate conversion module, a robotic arm motion planning module, a robotic arm control module, a mechanical claw control module, a data recording and analysis module and a user interaction interface.

[0044] The device sorting system uses the following steps to sort devices:

[0045] Step 1: The camera captures the device image and transmits it to the server;

[0046] In step 2, the image processing and recognition module in the server processes the collected device images to identify the device type and location. The image processing and recognition module uses the YOLOv5 algorithm to recognize the device images. Other recognition methods include convolutional neural network (CNN) architectures, such as ResNet, Inception, MobileNet, etc. These networks have their own advantages in different performance and accuracy. At the same time, attention mechanisms such as SENet (Squeeze-and-Excitation Networks) or CBAM (Convolutional Block Attention Module) can also be introduced to enhance the model's ability to recognize key features in the image.

[0047] Step 3: The coordinate conversion module converts the recognition result obtained in step 2 from image coordinates to actual space coordinates;

[0048] Step 4: The robot arm motion planning module calculates the joint angles and motion paths of the robot arm based on the actual spatial coordinates obtained in step 3;

[0049] Step 5: The robot arm control module controls the robot arm to move above the device according to the calculation result of step 4; the robot claw control module receives the signal and performs the grasping action;

[0050] Step 6: The robot arm moves the device to the designated classification area and releases it with the robot claw;

[0051] Step 7: The user interaction interface displays the sorting process and system status in real time and receives user instructions;

[0052] Step 8: The data recording and analysis module records the sorting results and provides a basis for performance optimization. In view of the actual needs of device sorting, this embodiment selects crawled images and actually collected images, and uses the labeling tool Labelimg to label them to form a device classification data set, including images and labeling files. Some data is shown in the figure below. The data set formed by cleaning and labeling data can effectively ensure data quality, help improve the model's recognition accuracy for specific device types, and enhance the adaptability and practicality of the system.

[0053] In step 2, the image processing and recognition module adopts the YOLOv5 deep learning algorithm and uses a custom training set to train the model to identify different types of devices, and can output the device category, position / pixel coordinates and size information; in step 3, the coordinate conversion module adopts Zhang Zhengyou calibration method to calibrate the camera's internal and external parameters, determine the conversion relationship between the camera coordinate system and the world coordinate system, and obtain the three-dimensional space coordinates of the device in combination with the depth image to realize the conversion from pixel coordinates to actual space coordinates; in step 4, the robot arm motion planning module adopts inverse kinematics calculation to accurately calculate the required joint angle according to the posture of the robot arm end effector and the position of the target object.

[0054] The working process can be briefly described as follows: first, place the device in a designated box or frame, use the depth camera to obtain the 2D image and 3D depth information of the device, use the image processing and recognition module to process the image, identify the device type, and determine the position of the device in the world coordinate system; start the robotic arm to obtain the angles of each joint through inverse kinematics calculation based on the recognition result and coordinate information; the robotic arm moves to the top of the device, and the mechanical claw adjusts the opening and closing degree according to the size and shape of the device to grab the device; the device is moved to the corresponding classification area to complete the sorting.

Claims

1. A device sorting system based on image recognition technology, comprising a robotic arm, characterized in that: The robotic arm is a multi-degree-of-freedom robotic arm, which is provided with a robotic claw and a camera. It also includes a server connected to the robotic arm, the robotic claw and the camera. The server is provided with an image processing and recognition module, a coordinate conversion module, a robotic arm motion planning module, a robotic arm control module, a robotic claw control module, a data recording and analysis module and a user interaction interface.

2. The device sorting system according to claim 1, characterized in that: The following steps are used to sort the devices: Step 1: The camera captures the device image and transmits it to the server; Step 2: The image processing and recognition module in the server processes the collected device image to identify the device type and location; Step 3: The coordinate conversion module converts the recognition result obtained in step 2 from image coordinates to actual space coordinates; Step 4: The robot arm motion planning module calculates the joint angles and motion paths of the robot arm based on the actual spatial coordinates obtained in step 3; Step 5: The robot arm control module controls the robot arm to move above the device according to the calculation result of step 4; the robot claw control module receives the signal and performs the grasping action; Step 6: The robot arm moves the device to the designated classification area and releases it with the robot claw; Step 7: The user interaction interface displays the sorting process and system status in real time and receives user instructions; Step 8: The data recording and analysis module records the sorting results and provides a basis for performance optimization.

3. The device sorting system according to claim 2, characterized in that: In step 2, the image processing and recognition module uses the YOLOv5 deep learning algorithm and uses a custom training set to train the model to identify different types of devices and output the device category, location / pixel coordinates, and size information.

4. The device sorting system according to claim 2, characterized in that: In step three, the coordinate conversion module uses Zhang Zhengyou calibration method to calibrate the internal and external parameters of the camera, determine the conversion relationship between the camera coordinate system and the world coordinate system, and obtain the three-dimensional space coordinates of the device in combination with the depth image to realize the conversion from pixel coordinates to actual space coordinates.

5. The device sorting system according to claim 2, characterized in that: In step 4, the robot arm motion planning module uses inverse kinematics calculation to accurately calculate the required joint angles based on the posture of the robot arm end effector and the position of the target object.

Images