Automatic inspection robot for machining tool based on vision active guidance

Abstract

The invention belongs to the field of robot technology and machine vision, and discloses an automatic inspection robot for a machining tool based on vision active guidance. A high-quality machining tool image data set is manufactured, a tool recognition deep learning model based on the YOLO algorithm is trained, and autonomous recognition and real-time space positioning of a to-be-detected machining tool are completed through a binocular camera; the automatic inspection of the machining tool is realized by combining with the inspection path planning of the follow-up clamping robot of the AGV. According to the method provided by the invention, the YOLO visual identification technology and the binocular measurement technology are combined, the functions of autonomous identification of the to-be-detected cutter and real-time sensing of the target position can be quickly, accurately and stably realized, the automation and unmanned degree of the cutter inspection process can be further improved through the automatic inspection robot, and the machining and detection efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot technology and machine vision, and in particular relates to technologies such as visual recognition, visual positioning and visual servoing, and is specifically a robot for automatically inspecting machining tools based on active visual guidance. Background Art

[0002] As my country continues to develop, the aerospace field has become increasingly important as a frontier of scientific and technological innovation. High-end equipment such as large aircraft and strategic weapons are at the core of the industrial chain and are the main battlefield and commanding heights of economic and defense competition among major countries. The parts required in the aerospace field usually have complex geometric shapes, large thin-walled structures, and tiny precision components. The processing of the above-mentioned complex parts not only requires extremely high precision, but also needs to cope with a series of challenges such as high material hardness and complex processing paths. The working state of the machining tool directly affects the machining quality of the parts. Therefore, the full life cycle management of the machining tool is the key to ensuring the healthy working state of the machining tool.

[0003] In the process of tool life cycle management, the geometric parameters and wear status of the tool need to be measured before and after processing. At present, manual tool inspection has problems such as low efficiency and non-standardized processes, which seriously slows down the production rhythm. In order to improve the efficiency and intelligence of inspection, the automatic tool inspection technology based on mobile robots has become a hot research direction in intelligent manufacturing.

[0004] The "Visual Inspection Management System and Method for Processing Tools" (Patent No. CN201810906007.8) can obtain the time interval between each tool leaving the tool carrier each time it is out of the warehouse and being placed on the tool carrier when it is put into the warehouse, and store the value of the time interval in the management record information corresponding to each tool; use visual inspection to identify the identity code of the corresponding tool, and obtain the management record information of the corresponding tool; finally, realize the prediction of the tool life range, but the above technology does not take into account the automation of the inspection process of processing tools from the processing site to the inspection equipment. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the tool use station and the inspection station are separated in the processing site, and the tool is manually picked up and sent for inspection, with a low degree of automation, which seriously slows down the production cycle. The present invention invents an automatic tool identification and clamping system based on an on-site robot. The system creates a processing tool training set and uses the YOLOv8 algorithm for training to achieve real-time identification of the tool to be tested; uses binocular vision distance measurement to achieve spatial in-situ perception of the tool to be tested; and plans the clamping and inspection process to achieve efficient detection of processing tools.

[0006] The technical solution of the present invention is as follows:

[0007] A robot for automatic inspection of machining tools based on active vision guidance is used to create a high-quality machining tool image dataset, train a tool recognition deep learning model based on the YOLO algorithm, and complete the autonomous recognition and real-time spatial positioning of the machining tools to be tested through a binocular camera; the automatic inspection of machining tools is realized by combining the inspection path planning of the on-site gripping robot of the AGV trolley. The specific steps are as follows:

[0008] The first step is to design and build a vision-guided automatic inspection robot for machining tools;

[0009] Build an automatic inspection robot for machining tools, including an AGV trolley 1, an industrial computer 2, a 5G transmission module 3, an electric gripper 4, an end fixture 5, a binocular camera 6, and a robotic arm 7; the binocular camera 6 and the electric gripper 4 are mechanically connected to the end fixture 5, and the end fixture 5 is then connected to the end of the robotic arm 7, the binocular camera 6 is connected to the industrial computer 2 via USB, the electric gripper 4 is controlled through the I / O of the robotic arm 7, the robotic arm 7 and the industrial computer 2 communicate wirelessly via 5G TCP / IP to form a visual guidance and gripping unit; the base of the robotic arm 7 is mechanically connected to the AGV trolley 1, and the AGV trolley 1 and the industrial computer 1 communicate remotely via 5GTCP / IP to form an inspection system;

[0010] During the inspection process, the visual guidance and gripping unit moves with the AGV 1; during the gripping process, the binocular camera 6 and the electric gripper 4 move with the robotic arm 7; after the inspection instruction is issued, the AGV 1 automatically moves to the gripping target area, and then the binocular camera 6 identifies the tool to be tested and returns its spatial position to the industrial computer 2. The industrial computer 2 converts the spatial position from the camera coordinate system to the robotic arm base coordinate system, and controls the robotic arm 7 to move the electric gripper 4 to the gripping position. After confirming the clamping, the electric gripper 4 returns a signal to the industrial computer 2. The industrial computer 2 controls the AGV 1 to move to the placement target area, completing the real-time autonomous recognition of the tool, in-situ spatial perception and automatic gripping and inspection process;

[0011] The second step is to build a complex structure tool recognition model based on deep learning;

[0012] A dataset of 1,500 high-quality images was collected. The dataset meets the following conditions: (1) To ensure the diversity of image resolution in the dataset, four resolutions of 720P, 1080P, 2K, and 4K were collected. (2) To ensure the diversity of image viewing angles in the dataset, the image collection range is from 45° to 45°, with a gradient of 2° and a horizontal viewing angle as the benchmark. (3) To ensure the diversity of distances, the image collection range is from 0.2m to 4m, with a gradient of 0.1m and a benchmark of 0.2m. The proportion of images with a resolution of 720P in the dataset is 21. %, the proportion of images with a resolution of 1080P is 27%, the proportion of images with a resolution of 2K is 27%, the proportion of images with a resolution of 4K is 25%, and the number of angle gradient and distance gradient images is equal. Finally, the Labelimg plug-in is used to mark and classify the types of tools and tool handles in the image; the collected images and marking information are divided into training set, test set and verification set in a ratio of 3:1:1, among which the training set must ensure that it covers the largest sample size and has diversity, and the verification set and test set must ensure that their image backgrounds are complex and difficult to identify;

[0013] In order to ensure the real-time and accurate recognition performance of the tool recognition model, the YOLO v8x pre-trained model is used to perform deep learning training on the data set in the CUDA 12.1 environment. By setting the training parameters such as the number of pre-training training epochs, the number of images processed in each batch (batch), and the number of working threads (workers), the recognition effect and training time are balanced to obtain the tool recognition model; after the training is completed, the weight files of the two tool recognition models "best.pt" and "last.pt" are generated. The latter is the last file saved in the model training process. It is used to continue training from the breakpoint of the last training, or for reasoning and evaluation of the tool recognition model; the former is the model weight file with the best performance on the training set or test set. It is used for subsequent visual recognition. The image to be recognized is input, and it will divide the image into a grid form. The recognized tool handle and the center position of the tool are output through probability calculation, and the confidence level of the current prediction result is given:

[0014]

[0015] Among them, Class represents the classification result, i is the category, Object is the detected object, truth is the true value, pred is the predicted value, and IOU is the overlap level between the predicted result and the true value; P(Class i |Object) is the probability that the target object belongs to a specific category; P(Object) is the probability that the target object appears in the image, which is specifically expressed as the confidence of the model detecting the existence of an object in a certain area; P(Class i) is the probability of a specific category appearing in the image;

[0016] The third step is to build a binocular vision-based algorithm for sensing the spatial position of the target tool and accurately clamping the tool;

[0017] In the constructed automatic inspection robot for machining tools, the binocular camera 6 is the main visual information perception element, which is composed of two left and right cameras with different viewing angles. The relative relationship between the two coordinate systems is as follows:

[0018]

[0019] Among them, x r ,y r 、z r is the coordinate of the object in the right camera coordinate system, x, y, z are the positions of the object in the left camera coordinate system, r i They represent the axis directions of the left camera coordinate system mapped to the right camera coordinate system, t i The translation of the left camera coordinate system in each axis of the right camera coordinate system. The matrix R and matrix T represent the left lens coordinate system O-xyz and the right camera coordinate system O r -x r y r z r The rotation matrix and translation vector of the transformation relationship between them are directly obtained through the camera calibration process;

[0020] During the recognition process, the binocular camera 6 collects images simultaneously, and the pixel spatial positions of the tool and the tool handle in the two sets of images are obtained through the tool recognition model. The camera calibration result is compared with the parallax of the two sets of images, and then the optical triangulation method is used to further solve the real three-dimensional coordinates (x cam ,y cam , z cam ):

[0021]

[0022] Among them, x and y are the pixel coordinates of the object in the left camera image plane, z is the depth of the object in the left camera coordinate system, and f l 、f r Respectively represent the focal length of the left and right cameras, X l , Y l is the coordinate of the object in the left camera coordinate system, X r , Y r is the coordinate of the object in the right camera coordinate system;

[0023] Since the gripping process needs to perform inverse kinematics calculation in the robot base coordinate system, it is also necessary to obtain the fixed relative relationship T between the camera coordinate system and the robot end coordinate system through hand-eye calibration, and obtain the transformation relationship T between the robot end coordinate system and the robot base coordinate system through DH table solution, that is, the target tool space position (x cam ,y cam , z cam ) is converted to the robot base coordinate system:

[0024]

[0025] in, R P g(i) is the coordinate of the center point of the end gripper in each posture in the robot base coordinate system; is the transformation matrix between the manipulator end coordinate system and the base coordinate system in each posture, which is given by the DH table of the manipulator; is the transformation matrix between the camera coordinate system and the robot arm end coordinate system and the end coordinate system. Since the “eye on hand” installation method is adopted, the camera and the robot arm end remain relatively fixed in each posture, which is obtained through hand-eye calibration; C P g is the coordinate of the center point of the end gripper in the camera coordinate system, which is obtained by binocular measurement;

[0026] In order to accurately grasp the target tool, it is only necessary to move the robot arm 7 so that the center coordinate of the electric gripper 4 R P g(i) and target tool coordinates R P obj(i) Just overlap;

[0027] Step 4: Set up the robot’s automatic inspection process;

[0028] The automatic inspection process of machining tools is divided into six parts: scheduling, shifting, searching, clamping, inspection, and homing. First, the robot stops at the initial site. After the user issues a scheduling command in the control software, the command is remotely transmitted to the AGV 1 and the robot arm 7 through the 5G transmission module 3, controlling the robot arm 7 to move to the shift protection position, and controlling the AGV 1 to shift to the clamping search site. Then, at the search site, a 360° target search action is performed at the end of the robot arm 7. After the binocular camera 6 finds that there is a machining tool in the field of view, the robot arm 7 stops moving, and controls the robot arm 7 to move to the fine positioning position according to the rough positioning result. In the fine positioning position, the target machining tool is obtained. The robot arm 7 detects the tool in the space and controls the end center of the robot arm 7 to reach the clamping position, and controls the electric gripper 4 to hold and clamp the tool through IO; after the electric gripper 4 returns the confirmation signal of the completion of clamping, the robot arm 7 returns to the shift protection position again, and controls the AGV trolley 1 to shift to the inspection booth, and the robot arm 7 places the clamped tool in the tool library to be inspected to complete the inspection process; finally, after confirming that the placement is completed, the AGV trolley 1 returns to the initial site, and if the power is lower than the set standard value, it automatically goes to the charging pile position for charging; through the above process, the automatic, accurate and efficient on-site inspection process of the target processing tool from the storage position to the tool inspection center and the tool storage center is realized.

[0029] Beneficial effects of the invention: The method proposed in the invention combines YOLO visual recognition technology with binocular measurement technology, which can quickly, accurately and stably realize the autonomous identification of the tool to be tested and the real-time perception of the target position. The automatic inspection robot can further improve the automation and unmanned degree of the tool inspection process, and improve the processing and inspection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the on-site inspection system for processing tools.

[0031] Figure 2 This is the flow chart for inspection of processing tools.

[0032] Figure 3 Coordinate transformation diagram of the object to be measured.

[0033] In the picture: 1-AGV, 2-industrial computer, 3-5G transmission module, 4-electric gripper, 5-end fixture, 6-binocular camera, 7-robotic arm. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is described in detail below in conjunction with the technical scheme and the accompanying drawings.

[0035] The first step is to design and build a tool automatic inspection robot with active visual guidance;

[0036] like Figure 1As shown, this example uses the AUBO i5H collaborative robot, which has a maximum load of 5kg, an arm span of 1008mm, and a working accuracy of ±0.02mm; the Dahuan Robot AG-160-95 electric gripper is used, which has a clamping force of 45N to 160N and a repeatability accuracy of ±0.02mm; the binocular measurement system uses the ZED 2i binocular camera, which has a focal length of 2.12mm, a sensor of 1 / 3” 4MP CMOS, and a resolution of 2688×1520;

[0037] The system uses an industrial computer as the control terminal. The binocular camera returns the identified three-dimensional coordinates of the tool handle to the computer through the USB2.0 interface. After compensation at the control terminal, the compensated coordinates are output to the control cabinet of the AUBO robot through the 5G transmission module. The control cabinet controls the end of the robotic arm to move to the specified position. Finally, the AUBO robot is connected to the electric gripper through an 8PIN aviation plug to realize IO control and thus achieve target gripping.

[0038] The second step is to build a complex structure tool recognition model based on deep learning;

[0039] The latest dataset used in this example contains 1727 images of complex step micro-tooth drills in the training set, 116 images in the test set, and 111 images in the validation set, totaling 1727 images, including two labels: drill_bit (tool) and tool_holder (tool holder), and the number of training rounds is 300 rounds;

[0040] To ensure the diversity of image viewing angles in the dataset, the image acquisition range is from 45° looking down to 45° looking up, with a gradient of 2°. To ensure the diversity of distances, the image acquisition range is from 0.2m to 4m, with a gradient of 0.1m, to ensure that the dataset covers the entire working area. To enhance the generalization of the training model, the dataset images have a variety of complex scenes, with multiple shooting tools such as SONY cameras, ZED 2i binocular cameras, and mobile phone lenses, and multiple resolutions such as 720P, 1080P, 2K, and 4K.

[0041] The environment configuration before training is shown in Table 1, and the training parameter settings are shown in Table 2:

[0042] Table 1 Configuration of YOLO environment related software versions

[0043]

[0044] Table 2 Training parameter settings

[0045]

[0046] The third step is to build a binocular vision-based algorithm for sensing the spatial position of the target tool and accurately clamping the tool;

[0047] The YOLO recognition program is combined with the binocular measurement program to obtain the center coordinates of the identification frame of the tool to be measured on the image, and the center coordinates are used as the landmark point. The three-dimensional space coordinates of the tool in the camera coordinate system are calculated by formulas (2) and (3). Finally, the robot arm is moved by formula (4) so ​​that the center coordinates of the end gripper coincide with the coordinates of the target tool, and the target tool is accurately clamped. The binocular measurement results of the tool to be measured are compared with the measurement results of the laser tracker. It can be obtained that the measurement accuracy of the binocular measurement system is 0.201mm.

[0048] Step 4: Set up the robot’s automatic inspection process;

[0049] like Figure 3 As shown, efficient and stable active inspection of machining tools can be achieved through the automatic inspection process of machining tools such as scheduling, shifting, searching, clamping, inspection, and homing.

Claims

1. A robot for automatic inspection of machining tools based on active vision guidance, characterized in that: Here are the steps: The first step is to design and build a vision-guided automatic inspection robot for machining tools; A processing tool automatic inspection robot is constructed, comprising an AGV trolley (1), an industrial computer (2), a 5G transmission module (3), an electric gripper (4), an end fixture (5), a binocular camera (6) and a robotic arm (7); the binocular camera (6) and the electric gripper (4) are fixedly connected to the end fixture (5) through a mechanical connection, and then the end fixture (5) is connected to the end of the robotic arm (7), the binocular camera (6) is connected to the industrial computer (2) through a USB, the electric gripper (4) is controlled through the I / O of the robotic arm (7), the robotic arm (7) and the industrial computer (2) communicate wirelessly through a 5G TCP / IP method, forming a visual guidance and gripping unit; the base of the robotic arm (7) is mechanically fixedly connected to the AGV trolley (1), and the AGV trolley (1) and the industrial computer 1 communicate remotely through a 5G TCP / IP method, forming an inspection system; During the inspection process, the visual guidance and gripping unit moves with the AGV trolley (1); during the gripping process, the binocular camera (6) and the electric gripper (4) move with the mechanical arm (7); after the inspection instruction is issued, the AGV trolley (1) automatically moves to the gripping target area, and then the binocular camera (6) identifies the tool to be tested and returns its spatial position to the industrial computer (2). The industrial computer (2) converts the spatial position from the camera coordinate system to the mechanical arm base coordinate system, and controls the mechanical arm (7) to move the electric gripper (4) to the gripping position. After confirming the clamping, the electric gripper (4) returns a signal to the industrial computer (2). The industrial computer (2) controls the AGV trolley (1) to move to the placement target area, completing the real-time autonomous recognition of the tool, in-situ spatial perception and automatic gripping and inspection process; The second step is to build a complex structure tool recognition model based on deep learning; A dataset of 1500 high-quality images was collected, and the dataset met the following conditions: (1) To ensure the diversity of image resolution in the dataset, four resolutions of 720P, 1080P, 2K, and 4K were collected respectively; (2) To ensure the diversity of image perspectives in the dataset, with 2° as the gradient and horizontal perspective as the benchmark, the image collection range was from 45° looking down to 45° looking up; (3) To ensure the diversity of distances, with 0.1m as the gradient and 0.2m as the benchmark, the image collection range was from 0.2m to 4m; the proportion of images with a resolution of 720P in the dataset was 21%, the proportion of images with a resolution of 1080P was 27%, the proportion of images with a resolution of 2K was 27%, and the proportion of images with a resolution of 4K was 25%. The number of images with gradients at each angle and distance was equal. Finally, the Labelimg plug-in was used to mark and classify the types of tools and tool handles in the images; the collected images and marking information were divided into training set, test set, and validation set in a ratio of 3:1:1; In the CUDA12.1 environment, the YOLO v8x pre-trained model is used to perform deep learning training on the data set. By setting the number of pre-training training times, the number of images processed in each batch, and the number of working threads, the recognition effect and training time are balanced to obtain the tool recognition model; after the training is completed, the weight files of the two tool recognition models "best.pt" and "last.pt" are generated. The latter is the last file saved in the model training process, which is used to continue training from the breakpoint of the last training, or for reasoning and evaluation of the tool recognition model; the former is the model weight file with the best performance on the training set or test set, which is used for subsequent visual recognition. The image to be recognized is input, and it will divide the image into a grid form. The recognized tool handle and the center position of the tool are output through probability calculation, and the confidence level of the current prediction result is given: Among them, Class represents the classification result, i is the category, Object is the detected object, truth is the true value, pred is the predicted value, and IOU is the overlap level between the predicted result and the true value; P(Class i |Object) is the probability that the target object belongs to a specific category; P(Object) is the probability that the target object appears in the image, which is specifically expressed as the confidence of the model detecting the existence of an object in a certain area; P(Class i ) is the probability of a specific category appearing in the image; The third step is to build a binocular vision-based algorithm for sensing the spatial position of the target tool and accurately clamping the tool; In the constructed automatic inspection robot for machining tools, the binocular camera (6) is the main visual information perception element. It is composed of two left and right cameras with different viewing angles. The relative relationship between the two coordinate systems is as follows: Among them, x r ,y r 、z r is the coordinate of the object in the right camera coordinate system, x, y, z are the positions of the object in the left camera coordinate system, r i They represent the axis directions of the left camera coordinate system mapped to the right camera coordinate system, t i The translation of the left camera coordinate system in each axis of the right camera coordinate system. The matrix R and matrix T represent the left lens coordinate system O-xyz and the right camera coordinate system O r -x r y r z r The rotation matrix and translation vector of the transformation relationship between them are directly obtained through the camera calibration process; During the recognition process, the binocular camera (6) simultaneously collects images, and the pixel spatial positions of the tool and the tool handle of the two sets of images are obtained respectively through the tool recognition model. The camera calibration result is compared with the parallax of the two sets of images, and then the optical triangulation method is used to further solve the real three-dimensional coordinates (x cam ,y cam , z cam ): Among them, x and y are the pixel coordinates of the object in the left camera image plane, z is the depth of the object in the left camera coordinate system, and f l 、f r Respectively represent the focal length of the left and right cameras, X l , Y l is the coordinate of the object in the left camera coordinate system, X r , Y r is the coordinate of the object in the right camera coordinate system; Since the gripping process needs to perform inverse kinematics calculation in the robot base coordinate system, it is also necessary to obtain the fixed relative relationship T between the camera coordinate system and the robot end coordinate system through hand-eye calibration, and obtain the transformation relationship T between the robot end coordinate system and the robot base coordinate system through DH table solution, that is, the target tool space position (x cam ,y cam , z cam ) is converted to the robot base coordinate system: in, R P g(i) is the coordinate of the center point of the end gripper in each posture in the robot base coordinate system; is the transformation matrix between the manipulator end coordinate system and the base coordinate system in each posture, which is given by the DH table of the manipulator; is the transformation matrix between the camera coordinate system, the robot arm end coordinate system and the end coordinate system. Since the "eye on hand" installation method is adopted, the camera and the robot arm end remain relatively fixed in each posture, which is obtained through hand-eye calibration; C P g is the coordinate of the center point of the end gripper in the camera coordinate system, which is obtained by binocular measurement; In order to accurately grasp the target tool, it is only necessary to move the mechanical arm (7) so that the center coordinate of the electric clamp (4) R P g(i) and target tool coordinates R P obj(i) Just overlap; Step 4: Set up the robot’s automatic inspection process; The automatic inspection process of machining tools is divided into six parts: scheduling, shifting, searching, clamping, inspection, and homing. First, the robot stops at the initial site. After the user issues a scheduling command in the control software, the command is remotely transmitted to the AGV car (1) and the robot arm (7) through the 5G transmission module (3), and the robot arm (7) is controlled to move to the shift protection posture, and the AGV car (1) is controlled to shift to the clamping search site. Then, at the search site, the end of the robot arm (7) performs a 360° target search action. After the binocular camera (6) finds that there is a machining tool in the field of view, the robot arm (7) stops moving, and controls the robot arm (7) to move to the fine positioning posture according to the rough positioning result. In the fine positioning posture, the target machining is obtained. The tool spatial position is determined, and the end center of the robot arm (7) is controlled to reach the gripping position, and the electric gripper (4) is controlled by IO to grip and grip the tool; after the electric gripper (4) returns a signal to confirm the gripping completion, the robot arm (7) returns to the shift protection position again, and controls the AGV trolley (1) to shift to the inspection booth, and the robot arm (7) places the gripped tool in the tool library to be inspected, completing the inspection process; finally, after confirming that the placement is completed, the AGV trolley (1) returns to the initial station, and if the power is lower than the set standard value, it automatically goes to the charging pile position for charging; through the above process, the automatic, accurate and efficient on-site inspection process of the target processing tool from the storage position to the tool inspection center and the tool storage center is realized.

Images