A robot vision system automatic alignment focusing method for tool damage detection

Abstract

The application discloses a kind of robot vision systems automatic alignment focusing methods for tool damage detection, the described include: the automatic alignment algorithm of intelligent region of interest (ROI) of fusion YOLOv5 network and improved Laplacian automatic focusing algorithm are proposed.First, search tool position and defogging denoising.Then, with tool center point as key feature, through the optimization of multi-point distribution movement and weighted correction least square method, the alignment pose of robot arm end is calculated.Finally, adaptive dynamic screening ROI focusing window, take the gradient square sum after the convolution operation of this area and Laplace operator as definition value, to determine the focusing pose of robot arm, finally get high-quality tool damage image.The method overcomes the limitations of traditional methods in multi-depth oil dirt dust harsh working conditions and the complexity of the measured target profile, sensitive to reflect definition mutation, improve focusing sensitivity, reduce alignment center point error, effectively exclude background and oil dirt dust interference, improve the efficiency and flexibility of tool damage vision detection system.

Description

Technical Field

[0001] The present invention belongs to the technical field of tool damage detection, and more specifically, relates to an automatic alignment and focusing method of a robot vision system for tool damage detection. Background Art

[0002] High-end CNC machine tools are core processing equipment in the cutting manufacturing industry. Cutting tools, the direct executors of mechanical cutting operations, inevitably experience wear, significantly impacting product quality, production efficiency, and safety. In recent years, the rapid development of computer informatization has fostered the interdisciplinary convergence of machine vision and tool wear detection. Modern machine vision-based tool damage detection technologies offer advantages such as high speed and non-contact detection, providing an effective means for tool health management and life prediction. Furthermore, robot-based tool visual inspection systems can be installed on automated guided mobile devices and applied to production lines comprised of multiple machine tools, addressing the difficulties associated with retrofitting mid-range and high-end machine tools, low efficiency, and poor visual inspection performance. However, in CNC machining scenarios, visual inspection presents challenges such as tool center deviation from the center of view and blurred tool images due to multiple depths of field between the machine tool spindle and the clamped tool, unstable robotic arms, and the inability to adaptively calibrate the position. Traditional solutions rely on manual visual observation, manually adjusting fixed-focus lenses, or moving robotic arms for focus and alignment. These methods are inflexible, time-consuming, and difficult to automate, significantly impacting the efficiency of intelligent tool damage detection. Therefore, there is an urgent need for an automatic alignment and focusing method for robot vision systems for tool damage detection to achieve adaptive acquisition of high-quality damage image samples and improve the efficiency of intelligent diagnosis of tool damage.

[0003] Chinese Patent Publication No. CN114367996A discloses a tool damage in-situ detection and tool changing robot, which combines tool visual in-situ detection with tool changing. However, the vision module based on the robotic arm ignores the research on alignment and focusing. Chinese Patent Publication No. CN103586740A discloses a micro-precision machining tool working morphology in-situ detection device and method. This method uses a robot to clamp a CCD camera to detect the tool working morphology on the machine. However, when obtaining different tool surface images, it is necessary to repeatedly rotate the robot or adjust the position of the tool being measured to focus on a clear tool image, resulting in a low degree of automation. ; Chinese Patent Publication No. CN113843659A discloses a spindle-integrated tool damage visual detection system, but the machine vision detection subsystem has difficulty in calculating the focus alignment posture for different tools and harsh working conditions; Chinese Patent Publication No. CN117140188A discloses an online prediction method and device for the remaining life of CNC lathe tools, which realizes on-machine tool status visual detection, but the camera of this method is fixed at one end of the rotating block, the visual system has poor flexibility and low acquisition efficiency, and the automatic alignment of the imaging center and the tool center is not considered, and the tool edge information will be lost. In summary, the automatic focus alignment method in the field of tool damage visual detection still has certain shortcomings and is difficult to adapt to the needs of rapid acquisition of high-quality visual images in multi-depth industrial scenarios. Therefore, there is an urgent need for a set of automatic alignment and focusing methods for robot vision systems for tool damage detection, which can be applied to objects with complex contour features and harsh working conditions, eliminate interference from irrelevant background factors, quickly focus and capture clear tool images with complete damage information, and realize adaptive acquisition of high-quality damage image samples. Summary of the Invention

[0004] The purpose of the present invention is to provide an automatic alignment and focusing method for a robot vision system for tool damage detection. The method integrates the YOLOv5 network intelligent region of interest (ROI), effectively eliminating background interference while quickly achieving clear and accurate acquisition of high-quality tool damage images. It solves the existing problem of on-machine tool damage visual detection in the field of manual adjustment of parameters of industrial cameras before detection to achieve alignment and focusing, resulting in a long time and poor flexibility. It effectively improves the efficiency and flexibility of the tool damage visual detection system.

[0005] The technical solutions for achieving the purpose of the present invention are:

[0006] A method for automatic alignment and focusing of a robot vision system for tool damage detection includes the following steps: uniformly numbering and grayscale processing the acquired tool images, and performing image annotation. The resulting images and corresponding classification files are randomly divided into a training set and a validation set (test set) in a 9:1 ratio. YOLOv5 model training is performed, and the optimal detection model is obtained through measures such as hyperparameter adjustment and increasing the number of iterations, resulting in a ROI window detection model.

[0007] A preprocessing module for the tool visual inspection system was constructed, including a coarse and fine step search layer, a defogging layer for wet cutting oil mist under harsh working conditions, and a noise reduction layer for dry cutting dust. When the robotic arm enters the machine tool, it performs a long-range, coarse step-by-step rapid search within the pre-defined spindle working area at four locations. It then fuzzily determines the two points containing the tool position. It then performs a fine search between two short-range locations with reduced step sizes until the tool center is within the field of view. The defogging layer uses a dark channel prior algorithm, and the noise reduction layer employs a median filter.

[0008] An automatic alignment module for multi-point distributed movement of the tool center is constructed. The tool center is detected and located using a ROI model. The alignment coordinates of the end-of-arm robot are calculated through multi-keypoint distribution and nonlinear weighted modified least squares optimization. The end-of-arm robot moves according to a specific multi-point distribution, obtaining the coordinates of multiple sets of feature points in the pixel coordinate system and the corresponding robot arm coordinates. A coordinate equation is established, in which polynomial local weights and ROI detection correction terms are added to the pixel point coordinates to adjust the edge tool center coordinates. The global weight of each point group is set to decrease nonlinearly in the radial direction, assigning smaller distance weights to groups of coordinate points that are off-center and therefore less reliable. The affine transformation matrix is ​​calculated, thereby deriving the coordinate change relationship between the pixel coordinate system and the end-effector coordinate system.

[0009] A tool vision image autofocus module was constructed based on a robotic arm and an improved Laplacian algorithm. Within the working height range, the robotic arm was moved along the machine tool spindle while the camera was simultaneously translated to read images and acquire pose information in real time. The image stream was preprocessed by reconstruction and frame statistics. Then, an improved Laplacian autofocus algorithm was employed to adaptively and dynamically select a window. The grayscale value of the tool feature window was used as input, and a convolution operation with the Laplace operator was performed to obtain the gradient value. The sum of the squared gradients of each pixel was taken as the clarity evaluation value. Finally, the optimal frame index was searched, the optimal frame ROI region was extracted, and the pose corresponding to this index was read. The end-arm was then controlled to move to the optimal pose to achieve autofocus.

[0010] Compared with the existing tool visual inspection and focusing method, the invention has the following significant advantages:

[0011] (1) This invention fully utilizes the flexibility of the robot arm and combines fine step search with an alignment and focusing algorithm to efficiently achieve adaptive calibration of the visual system's posture, targeting complex-profile tools and harsh industrial environments. At the same time, it reduces equipment costs, enhances the versatility of fixed-focus vision cameras in visual inspection, and reduces the impact of oil, dirt, and dust on high-quality images.

[0012] (2) The present invention selects the tool center point as the key feature and multi-point key coordinate according to local conditions, omitting the pre-calibration cost and significantly simplifying the robot vision system alignment steps, thereby improving alignment efficiency. By optimizing the coordinate relationship using the weighted modified least squares method, i.e., setting polynomial local weights, radial nonlinear global weights, and ROI detection correction terms, the alignment center point error is reduced, effectively preventing tool image distortion and blade edge information loss, making the image data more credible and usable, and improving the accuracy of visual quantitative detection.

[0013] (3) The improved Laplacian autofocus algorithm described in the present invention dynamically adjusts the focus window according to the specific characteristics of the tool, limits pixel-level calculations to the feature window, sensitively reflects sudden changes in clarity, effectively eliminates interference from background factors such as the machine tool spindle, reduces the amount of visual image processing calculations, greatly improves the focusing speed and sensitivity, and demonstrates excellent high-quality visual image acquisition performance in multi-depth of field industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of the automatic alignment and focusing method of a robot vision system for tool damage detection;

[0015] Figure 2 This is a physical picture of the tool damage visual detection system;

[0016] Figure 3 This is the architecture diagram of the automatic alignment and focusing method of the robot vision system for tool damage detection;

[0017] Figure 4 The tool ROI dynamic window image; Figure (a) shows the normal position, Figure (b) shows the tool imaging blurred position, and Figure (c) shows the tool being partially blocked;

[0018] Figure 5 is the local weight distribution map of weighted least squares method; DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] In order to realize the intelligent collection of tool damage visual images and improve the intelligence and automation level of tool damage detection, this embodiment provides an automatic alignment and focusing method of a robot vision system for tool damage detection, which specifically includes establishing an ROI window model, constructing a tool vision detection system preprocessing module, an automatic alignment module for multi-point distribution movement of the tool center, and an automatic focusing module for tool vision images.

[0021] First, a tool ROI window positioning detection dataset is established. The original images are collected by a robotic arm and an industrial camera. These data images include images in different situations such as clear and complete, blurred, underfocused, and overfocused. Data enhancement is performed by flipping, rotating, and other methods to expand the number of dataset samples. The tool image samples are labeled as Cutter, and the labeled tool dataset is divided into a training set and a validation set (test set) with a ratio of 9:1. The tool ROI window model is trained using the YOLOv5 single target detection network structure. The constructed dataset is selected to test the network performance. The number of training iterations is set to 100 and the hyperparameters are fine-tuned. The batch size is set to 16, and a unified image size of 640×640 and format is input. The COCO (Common Objects in Context) dataset pre-trained weights are used for training and testing. The center of the prior frame is calculated. If the predicted frame width and height are w and h respectively, the coordinates of the upper left corner are (x p ,y p ), the center coordinates are:

[0022]

[0023] A preprocessing module for the tool visual inspection system was constructed, comprising a coarse and fine step search layer, a defogging layer for wet cutting oil mist under harsh working conditions, and a denoising layer for dry cutting dust. The search layer consists of a coarse search and a fine search. When the robotic arm enters the machine tool, it conducts a rough and rapid search at a speed of 15-20 mm / s within the pre-defined spindle working area, at the lowest quartile of the non-interference area. It fuzzily determines the two points containing the tool position. It then performs a fine search at a speed of 6-8 mm / s, decreasing the step size, between the two points over a short distance until the tool center is within the field of view. The robotic arm then adjusts to the initial pose of the captured image. Repeated inspections can be performed to improve tool image recognition accuracy. The defogging layer uses a dark channel prior algorithm to estimate the oil mist concentration and image transmittance by searching the dark channel of the tool image. Defogging is then performed based on the transmittance. The denoising layer uses a median filter to set the grayscale value of a pixel in the tool image to the median of all pixels in its neighborhood, eliminating dust noise and effectively preserving tool edge information.

[0024] An automatic alignment module of multi-point distribution movement of tool center is constructed. In order to determine the mutual relationship between the three-dimensional geometric position of the tool center and the corresponding point in the image, the spindle clamped with the detected tool in the machine tool is taken as the reference object, n key positions are selected, and the n points should be uniformly distributed in the field of view, and n≥3 must be met. The larger the value of n is, the slower the speed is, but the smaller the alignment error is, and the specific value can be set according to the task requirement. The end of the mechanical arm is controlled to move from point 1 to point n, and the n end coordinates in the base coordinate system of the mechanical arm are recorded. The center point position is determined by using the ROI tool detection model, and the corresponding n tool feature center points in the pixel coordinate system are obtained.

[0025] The affine transformation of the industrial camera coordinate system to the mechanical arm coordinate system is solved, so that the coordinate change relationship of the pixel coordinate system and the end effector coordinate system is derived. The n sets of point coordinate relationship equations are established, the non-linear weighted least squares method is used for optimization, the pixel point coordinates are added with polynomial local weight and ROI detection correction term to adjust the edge tool center coordinates, and the global weight of each set of points is set to be radially non-linearly reduced, and the coordinate point groups with relatively low reliability are given smaller weight. x and E y The tool center pixel point set is represented as F x and F y The robot end point two-dimensional coordinate set is represented as F

[0026]

[0027] α i is the correction term of the i-th group of points in the x direction, β i is the correction term of the i-th group of points in the y direction. w i is the x direction weight of the i-th group of points, h i is the y direction weight of the i-th group of points, which is radially non-linearly increased:

[0028]

[0029] In the formula, r i is the distance of the i-th point from the center point of the field of view, θ i is the rotation angle, 0≤θ i ≤2π, a k , b k (k=1, 2, 3) are polynomial coefficients, which can be set according to the specific camera. For the central region, a1 and b1 play a major role; for the edge region, a2 and b2 play a major role; for extreme wide-angle lenses such as fisheye cameras, a3 and b3 terms can be added.G i is the global weight of the i-th set of coordinates, which is non-linearly reduced to weaken the influence of poor reliability sample points on the optimization effect:

[0030]

[0031] Where, the height and width of the picture are A, B, G respectively. i It's about r i The hyperbolic tangent function, when r i From 0 to monotonically decreasing in the range of r i =0, the maximum value is 1. The minimum value is close to 0.92, and the slope of the curve first approaches 0 and then decreases rapidly.

[0032] The optimized affine matrix H represents the coordinate transformation relationship between the industrial camera coordinate system and the end effector. H is a three-row and three-column matrix:

[0033]

[0034] Multiply the affine matrix H by the frame center pixel coordinate matrix to calculate the alignment pose and control the robot arm movement to align the tool center. The frame center pixel coordinate is Then the two-dimensional coordinates of the alignment pose are:

[0035]

[0036] A tool vision image autofocus module based on a robotic arm and an improved Laplacian algorithm was constructed. This module only requires controlling the movement of the robotic arm to achieve focus and is applicable to any end-of-line camera model. After alignment, the robotic arm is moved along the machine tool spindle within the initial and extreme height ranges, while the camera is simultaneously translated to read images and obtain pose information in real time. Because the resolution of the industrial camera used reaches tens of millions of pixels, in order to improve computational speed and image processing results, the reconstructed image size is 1440×960 during image stream preprocessing. Data parameters include the image stream container f = {f1, f2, …, f m}, where m is the total number of frames, which is related to the camera frame rate. The robot end pose data container p={p1,p2,…,p m}, index i satisfies 1≤i≤m, the best frame is the initial frame, the best frame index is initialized to 1, and the best frame score is initialized to 0.

[0037] An improved Laplacian autofocus algorithm is used to calculate the sharpness evaluation value of the current frame. This algorithm adapts to the dynamic selection of the ROI window and limits the pixel-level calculation to the feature window, eliminating the interference of irrelevant background information and effectively reducing the amount of focusing calculation and time. The grayscale value of the tool feature window is used as the ROI (x, y) input, and then convolved with the Laplace operator to obtain the gradient value. The sum of the squares of the gradients of each pixel is taken as the sharpness evaluation value, which sensitively reflects the sudden change in sharpness. As shown in the formula:

[0038]

[0039] Where L is the Laplace operator,

[0040] The clarity evaluation values ​​of consecutive frames are compared. If the current frame evaluation value is greater than the historical maximum evaluation value, the system updates the optimal frame index. When the traversed frame index is the largest, the search ends, a high-quality tool image is obtained, and the robot arm is controlled to move to the optimal position to complete automatic alignment and focus. The tool recognition prediction frame is enlarged and intercepted at a ratio of k = 1.03. The coordinates of the upper left and lower right corners of the ROI area are:

[0041]

[0042] This automated alignment and focusing method for a robot vision system for tool damage detection enables accurate and clear automated capture of tool images. Compared to traditional methods, this method offers greater real-time performance and accuracy, and can be used in complex industrial visual inspection scenarios with multiple depths of field. This improves the flexibility and efficiency of tool damage visual inspection and enhances the automation of tool damage detection on production lines.

[0043] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatic alignment and focusing of a robot vision system for tool damage detection, characterized in that: The following steps are involved: Step 1: Establish a tool ROI window positioning detection dataset; Step 1.1: Collect original images using a robotic arm and industrial camera, perform data augmentation through flipping, rotation, and other methods to expand the number of dataset samples; Step 1.2: Label the tool image samples and divide the labeled tool dataset into a training set, a validation set, and a test set; Step 2: Train the tool ROI window detection model based on YOLOv5; Step 2.1: Select the constructed dataset to test the YOLOv5 network performance, set the number of training iterations and fine-tune the hyperparameters, set the input image size and format to a uniform size, and use the pre-trained weights for training and testing; Step 2.2: Calculate the center pixel coordinates of the prediction box; Step 3: Construct a preprocessing module for the tool vision inspection system, including a coarse and fine step search layer, a demisting layer for wet cutting oil mist under harsh working conditions, and a noise reduction layer for dry cutting dust; Step 3.1: The search layer includes a coarse search and a fine search. Based on the tool ROI window detection model described in step 2, when the robot arm enters the machine tool, it sets a long-distance four-point coarse step size rapid movement search within the pre-set machine tool spindle working area. It fuzzy judges the two points containing the tool position, and then reduces the step size to perform a fine search between the two short-distance points until the tool center is in the field of view. The robot arm adjusts to the initial position of the captured image. Step 3.2: The defogging layer uses a dark channel prior algorithm to estimate the oil mist concentration and image transmittance by finding the dark channel of the tool image. Then, defogging is performed based on the transmittance. The denoising layer uses a median filter to set the grayscale value of the tool image pixel to the median of the grayscale values ​​of all pixels in its neighborhood, thereby removing dust noise and effectively retaining the tool edge information. Step 4: Construct an automatic alignment module for multi-point distribution movement of the tool center; Step 4.1: Using the tool being inspected, which is clamped by the spindle in the machine tool, as a reference object, select n key positions, control the end of the robotic arm to move from point 1 to point n, record the coordinates of the n end in the robotic arm base coordinate system, and obtain the coordinates of the corresponding n tool feature center points in the pixel coordinate system; Step 4.2: Establish the relationship equation of n groups of point coordinates and use weighted modified least squares optimization. Polynomial local weights and ROI detection correction terms are added to the pixel coordinates to adjust the edge tool center coordinates. The global weight of each group of points is set to decrease nonlinearly in the radial direction. The distance weight of the coordinate point group that is off-center and has relatively low reliability is given a smaller distance weight. Step 4.3: Calculate the affine matrix, multiply the affine matrix by the pixel coordinates of the center of the field of view, calculate the alignment pose, and control the robot arm to move to the alignment pose; Step 5: Build a tool vision image auto-focus module based on the robotic arm and the improved Laplacian algorithm; Step 5.1: This module only needs to control the movement of the robot arm to achieve end-camera focus. It can be used with any model of vision camera. Under the alignment posture, the robot arm moves along the machine tool spindle direction, reads the image in real time and obtains the posture information, reconstructs the image stream size and initializes the data parameters. Step 5.2: By detecting the tool feature target in the original image stream, the ROI window is adaptively and dynamically selected, and pixel-level calculations are limited to the feature window, eliminating interference from irrelevant background information, effectively reducing the amount of focusing calculations and time. Step 5.3: By performing a convolution operation on each pixel point with the Laplace operator frame by frame to obtain the gradient value, the sum of the squares of the gradients in the ROI window image area is taken as the clarity evaluation value, which sensitively reflects sudden changes in clarity. Step 5.4: Determine the optimal frame index corresponding to the maximum peak value of clarity, control the robotic arm to adjust to the optimal focus position, read the original image of the optimal frame index, and enlarge the tool ROI area to obtain a high-quality tool image.

2. The automatic alignment and focusing method of a robot vision system for tool damage detection according to claim 1, characterized in that: Step 3.1: Before searching, it is necessary to combine the obtained machine tool structure information and perform a rough search at the quartile points of the lowest plane in the non-interference area at a speed of 15-20 mm / s. Then, move to the two points containing the tool and perform a fine search at a speed of 6-8 mm / s. The detection can be repeated multiple times to improve the recognition accuracy of the tool image until the tool enters the field of view.

3. The automatic alignment and focusing method of a robot vision system for tool damage detection according to claim 1, characterized in that: In step 4.1, the number of calibration points is set to n. The n points should be evenly distributed in the field of view, and n ≥ 3. The larger the n value, the slower the speed but the smaller the alignment error. The specific value can be set according to the task requirements.

4. The method for automatic alignment and focusing of a robot vision system for tool damage detection according to claim 1, wherein: In step 4.2, E x and E y represents the pixel set of the tool center, F x and F y Represents the two-dimensional coordinate set of the robot end point, the coordinate relationship equation and weighted modified least squares optimization are as follows: α i is the correction term in the x direction of the i-th group of points, β i is the y-direction correction term for the i-th group of points, w i is the x-direction weight of the i-th group of points, h i is the y-direction weight of the i-th group of points, which increases radially and nonlinearly: Where r i is the distance between the i-th point and the center of the field of view, θ i is the rotation angle, 0≤θ i ≤2π, a k 、b k (k=1, 2, 3) are polynomial coefficients, which can be set according to the specific camera size. For the center area, a1 and b1 play a major role; for the edge area, a2 and b2 play a major role; for extreme wide-angle lenses such as fisheye cameras, a3 and b3 can be added, G i is the global weight of the i-th group of coordinate points, which decreases nonlinearly to weaken the influence of sample points with poor reliability on the optimization effect: Where, the height and width of the picture are A, B, G respectively. i It's about r i The hyperbolic tangent function, when r i From 0 to monotonically decreasing in the range of r i =0, the maximum value is 1. The minimum value is close to 0.92, and the slope of the curve first approaches 0 and then decreases rapidly.

5. The automatic alignment and focusing method of a robot vision system for tool damage detection according to claim 1, characterized in that: In step 4.3, the optimized affine matrix H represents the coordinate transformation relationship between the industrial camera coordinate system and the base coordinate system; H is a three-row and three-column matrix: The pixel coordinates of the center of the frame are Then the two-dimensional coordinates of the alignment pose are:

6. The automatic alignment and focusing method of a robot vision system for tool damage detection according to claim 1, characterized in that: In step 5.1, since the resolution of the industrial camera used reaches tens of millions of pixels, in order to improve the operation speed and image processing results, the image size is reconstructed in the image stream preprocessing. The data parameters include the image stream container f = {f1, f2, ..., f m }, where m is the total number of frames, which is related to the camera frame rate, and the robotic arm end pose data container p = {p1, p2, ..., p m }, index i satisfies 1≤i≤m, the best frame is the initial frame, the best frame index is initialized, and the best frame score is initialized.

7. The automatic alignment and focusing method of a robot vision system for tool damage detection according to claim 1, characterized in that: The improved Laplacian autofocus algorithm in step 5 refers to adaptively selecting a window, taking the grayscale value of the feature window as the input of ROI (x, y), and then performing convolution operation with the Laplace operator to obtain the gradient value, and taking the square sum of the gradient of each pixel as the evaluation value. Where L is the Laplace operator template,

Images