Spraying robot spray gun teaching method based on double-event camera

Abstract

The invention relates to a spraying robot spray gun teaching method based on a double-event camera. The method comprises the following steps: calibrating internal and external parameters of two event cameras; the spray gun is provided with at least four mark points, and the rigid offset from the mark points to the center point of a muzzle is obtained; calibrating a tool coordinate system T of the center point of the industrial robot spray gun muzzle; establishing a homogeneous transformation matrix from the world coordinate system W to the industrial robot base coordinate system B; teaching: obtaining event flows of left and right event cameras of the spray gun and time flows of the spray gun spraying paint discharge amount; according to the event flow, the pose of the muzzle center point of the spray gun under the world coordinate system W is obtained and converted into the pose of the muzzle center point under the industrial robot base coordinate system B; and a control program of the industrial robot is generated according to the time flow of the spraying paint discharge amount and the event flow of the muzzle center point pose. According to the method, the motion trail is reconstructed through the double-event camera, the control program of teaching is generated in combination with the spraying time flow, and high-precision reproduction of the spraying teaching process is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot spraying teaching, and particularly to a spraying robot spray gun teaching method based on a dual-event camera. Background Art

[0002] Aiming at the disadvantage of low efficiency of manual spraying, many spraying application scenarios have begun to use industrial robots to replace manual labor. Among them, some spraying application occasions have high requirements for the accuracy of the teaching trajectory and the precise control of the spraying effect. For example, in the process of manufacturing jeans, in order to achieve effects such as local bleaching, it is usually necessary to spray the clothes. The characteristics of clothes spraying are: usually local spraying, different spraying effects and different rendering patterns will be produced due to different poses of the spray gun and the gradually changing pressing force of the spray gun during the spraying process, which will affect the aesthetics of the finished clothes.

[0003] In order to balance the flexibility and high precision of teaching in such spraying applications, the latest method is to use a hand-held spray gun for teaching. Compared with teaching using a teach pendant and drag teaching, hand-held spray gun teaching not only eliminates programming, but also is more flexible in operation, but the spray gun needs to be improved.

[0004] At present, some technical solutions have been proposed by researchers for obtaining the pose of an industrial robot hand-held teaching device, but there are still deficiencies in the spraying industry with high-precision and precise control requirements. The invention patent with the application number 202211387771.1 proposes a method for obtaining the position information of the teaching handle by the tensile force value of the connecting rope. This method requires tightening the connecting rope during teaching and lacks flexibility; the invention patent with the application number 202010302166.4 records the pose of the spray gun through an inertial measurement unit IMU, and then uses a grating line array group as an auxiliary sensor to eliminate the cumulative error of the displacement data obtained by IMU integration. This solution requires the installation of a huge grating line array group, which is not only costly but also difficult to use and maintain in a spraying occasion; the invention patent with the application number 202110716031.7 obtains the pose of the spray gun during the spraying process by fusing IMU sampling data and UWB sampling data. This solution requires the deployment of multiple base stations, and can only achieve centimeter-level positioning accuracy in an indoor environment, which cannot meet the high-precision spraying requirements; the invention patents with the application numbers 201910385954.1 and 201910408931.8 provide a stereo vision-based solution. This method will have motion blur when the teaching spray gun moves quickly, and is prone to overexposure or underexposure in strong light or weak light scenarios, resulting in difficulty in reconstructing the motion trajectory. Summary of the Invention

[0005] The object of the present invention is to provide a teaching method for the spray gun of a spraying robot based on a dual-event camera in view of the deficiencies of the prior art. This teaching method obtains the movement trajectory of the spray gun through two event cameras, forms a trajectory event stream, and combines it with the spraying amount time stream of the spray gun to form a control method for controlling the spray gun, enabling the teaching to be reconstructed again.

[0006] A teaching method for the spray gun of a spraying robot based on a dual-event camera, comprising:

[0007] Step 1: Calibrate the internal and external parameters of two event cameras;

[0008] Step 2: The spray gun is provided with at least 4 marking points, and the rigid offset from the marking points to the center point of the gun muzzle is obtained;

[0009] Step 3: Obtain the homogeneous transformation matrix from the tool coordinate system T of the gun muzzle center to the base coordinate system B of the industrial robot and the homogeneous transformation matrix from the world coordinate system W to the base coordinate system B of the industrial robot

[0010] Step 4: Teaching: Obtain the event streams of the left and right event cameras of the spray gun, and the spraying paint discharge amount size time stream of the spray gun spraying parameters;

[0011] Step 5: According to the event stream, obtain the pose of the center point of the gun muzzle in the world coordinate system W, and convert it to the pose of the center point of the gun muzzle in the base coordinate system B of the industrial robot; form the time stream of the pose of the center point of the gun muzzle;

[0012] Step 6: Generate a control program for the industrial robot according to the spraying paint discharge amount size time stream and the time stream of the pose of the center point of the gun muzzle.

[0013] The dual-event cameras are respectively a left event camera and a right event camera. When performing Step 1, synchronously collect event streams for the calibration board, and respectively generate a series of grayscale images I L and I R ; The world coordinate system W is established on the calibration board. Obtain the coordinates of multiple calibration points on the calibration board and the coordinates of the calibration point images in images I L and I R , and then respectively obtain the internal and external parameters of the two event cameras through calculation.

[0014] The calibration board is an array of LED boards, distributed with multiple LEDs. The LEDs flash according to a set program. The left and right event cameras synchronously collect event streams for the calibration board; select images corresponding to different poses of the calibration board, and extract the 2D pixel coordinates of the center of the LED image in the image and its 3D coordinates P(X W , Y W , 0) in the world coordinate system;

[0015] Force the camera coordinate system of the left event camera to coincide exactly with the world coordinate system. Then, the extrinsic parameter matrix and offset of the left event camera are: R l = I, t l = 0; Set the intrinsic parameter matrix K l of the left event camera and the distortion coefficient D l ; Extract multiple sets of corresponding coordinate points: P(X W , Y W , 0), Two coordinate points in a group correspond to an LED and the image of the LED in the image. Based on the monocular camera projection model:

[0016]

[0017] Construct a linear equation system to solve the camera projection matrix and obtain the initial value of the intrinsic parameter K l . Among them: u l，i , v l，i represent the coordinates of the center of the i-th LED extracted in the image coordinate system; X l，Wi , Y l，Wi represent the coordinates of the LED in the world coordinate system.

[0018] Then, adopt the objective function of jointly minimizing the binocular reprojection error to obtain K l and D l :

[0019]

[0020] Among them, N is the number of calibration board postures used in the calibration process; M is the number of LED points extracted for each calibration board posture; is the 2D pixel coordinate of the center of the j-th LED in the i-th posture observed by the right event camera; P r，ij is the theoretical 3D coordinate of the center of the j-th LED in the i-th posture observed by the right event camera in the world coordinate system; p r，ij (K r , D r , R1, t1, P r，ij ) is the theoretical 2D coordinate after projecting the theoretical 3D coordinate P r，ij onto the image plane through the camera model.

[0021] Calibrate the intrinsic parameter matrix K r and the distortion coefficient D r of the right event camera, as well as the rotation matrix R1 and translation vector t1 relative to the left event camera.

[0022] Using multiple sets of 3D-2D corresponding points of the right event camera, that is, in the image obtained by the right event camera, extract the world coordinates of multiple calibration points (LEDs) and the image coordinates where the corresponding images are located.

[0023] Based on the monocular camera projection model:

[0024]

[0025] Construct a system of linear equations to solve the camera projection matrix and obtain the initial values of K r , R1, and t1. Among them, s represents the scale factor, u r，i , v r，i are the coordinates of the center of the image of a calibration point (LED) in the image generated by the right event camera, and X r，Wi , X r，Wi represent the coordinates of the calibration point (LED) in the world coordinate system.

[0026] Then, by jointly minimizing the binocular reprojection error objective function, obtain K r , D r , R1, and t1:

[0027]

[0028] Among them, is the 2D pixel coordinate of the center of the jth calibration point (LED) in the ith pose observed by the right event camera; P r，ij is the theoretical 3D coordinate of the center of the jth calibration point (LED) in the ith pose observed by the right event camera in the world coordinate system; p r，ij (K r , D r , R1, t1, P r，ij ) is the theoretical 2D coordinate after projecting the theoretical 3D coordinate P r，ij onto the image plane through the camera model.

[0029] When performing step 2, L marking points (L≥4) are set on the spray gun. The center point of the spray gun nozzle touches the origin of the world coordinate system W through multiple different poses (the center point of the spray gun nozzle coincides with the origin), and the left and right event cameras obtain event stream data for each marking point.

[0030] According to the event stream data of the marking points, extract the coordinates of the L marking points in the world coordinate system W under K poses; and the two-dimensional coordinates of the corresponding marking points in the image coordinate system; both K and L are natural numbers; obtain the two-dimensional coordinates of the marking points detected in the left and right event cameras and n represents the nth pose, where n = 1 - K; l represents the left image generated by the left event camera, r represents the right image generated by the right perspective camera, and i represents the ith marker point, where i = 1 - L;

[0031] According to the internal and external parameters of the dual-event camera, for each marker point, a projection equation is constructed and solved by the least squares method to convert the matched two-dimensional point coordinates into three-dimensional coordinates

[0032]

[0033] Calculate the rigid offset ΔP i :

[0034] In the nth pose, the rigid transformation matrix of the spray gun is R n , and the translation vector t n = 0;

[0035] Minimize the error objective function of the observed coordinates and the model predicted coordinates in all poses:

[0036]

[0037] Then the rigid offset ΔP from the ith marker point to the center point of the gun muzzle can be obtained i .

[0038] When performing step 3:

[0039] Set the direction of the tool coordinate system T of the center point of the spray gun muzzle to be consistent with the flange center coordinate system F. The industrial robot controls the center point of the spray gun muzzle to touch the origin of the world coordinate system W in different poses, and the industrial robot control system respectively records the poses T of the flange center in the base coordinate system B in 4 different poses F , and solve the homogeneous transformation matrix by the least squares method Then the homogeneous transformation matrix from the tool coordinate system T to the industrial robot base coordinate system B can be obtained,

[0040] where, is the homogeneous transformation matrix from the industrial robot flange center coordinate system F to the base coordinate system B.

[0041] Establish the homogeneous transformation matrix from the world coordinate system W to the industrial robot base coordinate system B It is directly obtained by using the industrial robot to control the center point of the spray gun muzzle to calibrate the user coordinate system for the calibration plate coordinate system (world coordinate system W);

[0042] Specifically, the industrial robot controls the movement of the spray gun so that the center of the spray gun nozzle touches the origin of the world coordinate system defined on the calibration plate to obtain the origin data; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the X-axis side of the calibration plate to obtain the movement data information in the positive X-axis direction; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the Y-axis side of the calibration plate to obtain the movement data information in the positive Y-axis direction; thus, the homogeneous transformation matrix can be calculated.

[0043] When performing step 4 for teaching:

[0044] Remove the spray gun from the industrial robot and hold the spray gun for spraying teaching.

[0045] Two event cameras collect event streams in real time during the teaching process, and at the same time, the pressure sensor in the spray gun collects pressure data streams.

[0046] When performing step 5:

[0047] Receive the event streams of the two event cameras and the needle displacement data sent by the spray gun controller Perform data processing: Align the event streams of the two event cameras in time and segment the event streams;

[0048] Based on each segment of the event streams corresponding to the left and right cameras, calculate the pose of the center point of the spray gun nozzle in the world coordinate system, including:

[0049] Receive the event streams of the two event cameras and the data on the amount of paint discharged by the spray gun controller controlling the nozzle to spray paint, and perform data processing: Align the event streams of the two event cameras in time and segment the event streams;

[0050] Obtain each segment of the event streams corresponding to the left and right cameras to calculate the pose of the center point of the spray gun nozzle in the world coordinate system, including: Preprocess the event streams: First, perform time window filtering: Use the earlier event timestamp of the two event cameras as the starting point of the window, set the time window Δt = 1ms, for each segment of the event stream E i for each event e k in it, count the number of events in its spatial neighborhood (3×3 pixels) within the time window [t k -Δt, t k +Δt]. If the number is greater than or equal to 5, retain the event, and finally obtain the new event stream E f ;

[0051] Then divide the event stream E f into candidate clusters of L marked points:

[0052] Randomly select L events e1, e2,......, e LAs the initial clustering centers, for the event stream E f For each data point e i in it, calculate its distances from the L clustering centers. According to the distance values, assign e i to the cluster to which the nearest clustering center belongs.

[0053] For each formed cluster, calculate the mean of all events within the cluster, and take the e i nearest to the mean as the new clustering center, and then repeat the three steps of recalculating the distance values, assigning to clusters, and updating the clustering centers until the clustering centers no longer change.

[0054] Perform weighted averaging on the n event coordinates within each candidate cluster of the marked points:

[0055]

[0056] Where, is the weighted value of the events in the left camera event stream, and d i is the average distance from this event to other events within the cluster;

[0057] Calculate the two-dimensional coordinates and of the same marked point in the left and right event cameras, and then use the calibration parameters of the dual event camera to solve through the least squares method to convert the two-dimensional coordinate pair of the matching points into three-dimensional coordinates

[0058]

[0059] Calculate the pose of the spray gun:

[0060] Minimize the error objective function of the observed coordinates and the model predicted coordinates of the marked points obtained above:

[0061]

[0062] Then R and t can be obtained, where: the translation vector t is the current position value of the center point of the spray gun muzzle, and the rotation matrix R is the current pose of the spray gun in the world coordinate system;

[0063] Convert the pose of the center point of the spray gun muzzle in the world coordinate system to the pose of the center point of the muzzle in the base coordinate system of the industrial robot: Obtain the time stream of the pose of the center point of the muzzle.

[0064] Finally, generate the control program of the industrial robot according to the time stream of the paint discharge amount and the time stream of the pose of the center point of the muzzle.

[0065] Advantages of the present invention: The present invention reconstructs the motion trajectory through a dual-event camera and generates a teaching control program by combining the time flow of spraying, enabling the reconstruction of the teaching trajectory. Description of the Drawings

[0066] Figure 1 It is a working schematic diagram of the teaching in this embodiment.

[0067] Figure 2 It is a structural schematic diagram of the spray gun in this embodiment.

[0068] Reference Signs:

[0069] 01 - spray gun; 02 - event camera 1; 03 - event camera 2; 04 - calibration board; 05 - industrial control computer; 06 - industrial robot control system; 07 - industrial robot; 1 - gun body; 2 - gun needle; 3 - servo motor; 4 - connector; 5 - handle trigger; 6 - pressure sensor; 7 - controller; 8 - communication module; 9 - button; 10 - 15: marking points. Detailed Embodiment

[0070] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer, the following further details the present application in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0072] It should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present application.

[0073] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality" means two or more unless otherwise specifically defined.

[0074] The present invention will be described in detail below with reference to the accompanying drawings. As Figures 1 to 2 shown.

[0075] Example 1: Refer to Figures 1 to 2 ; A spray gun teaching method for a spraying robot based on a dual-event camera, including:

[0076] Step 1: Calibrate the internal and external parameters of two event cameras;

[0077] Step 2: Set at least 4 marking points on the spray gun, and obtain the rigid offset from the marking point to the center point of the gun muzzle;

[0078] Step 3: Obtain the homogeneous transformation matrix from the tool coordinate system T of the gun muzzle to the base coordinate system B of the industrial robot and the homogeneous transformation matrix from the world coordinate system W to the base coordinate system B of the industrial robot

[0079] Step 4: Teaching: Obtain the event streams of the left and right event cameras of the spray gun, and the time stream of the size of the spray coating discharge amount of the spray gun spraying parameters;

[0080] Step 5: According to the event stream, obtain the pose of the center point of the spray gun muzzle in the camera coordinate system W, and convert it to the pose of the center point of the gun muzzle in the base coordinate system B of the industrial robot; obtain the time stream of the pose of the center point of the gun muzzle;

[0081] Step 6: Generate the control program of the industrial robot according to the time stream of the size of the spray coating discharge amount and the time stream of the pose of the center point of the gun muzzle.

[0082] This embodiment is based on a dual-event camera, calibrates the spray gun, obtains the time trajectory of the spray gun during teaching, and the time stream of the size of the spray coating discharge amount sprayed by the center point of the gun muzzle; combining the time trajectory and the time stream of the size of the spray coating discharge amount can realize the reproduction of teaching for spray gun control. The time stream of the size of the spray coating discharge amount is not directly the size of the coating discharge amount, but can be parameters related to the size of the spray coating discharge amount, such as pressure parameters, the moving distance of the needle head, etc., which are proportional to the size of the spray coating discharge amount.

[0083] When performing Step 1, refer to Figure 1 , two event cameras 02 and 03 are fixedly spaced on the support frame, the support frame is arranged on the industrial control computer 05, the calibration plate 04 is arranged in the spraying space, and the calibration plate can be selected as a black and white board, a checkerboard, etc.; in this embodiment, the calibration plate is selected as an array of LED boards, and the LEDs are calibration points; the LEDs on the LED board are spaced, and the LED lighting mode is preferably random flashing; the lighting and extinguishing mode of the LED board can be controlled through a programming program; the calibration plate 04 is located on one side of the support frame, and the calibration plate 04 can be placed horizontally.

[0084] During calibration, the left and right event cameras 02 and 03 synchronously acquire event streams for the calibration board 04; for different poses (light-emitting patterns) of the calibration board, a series of grayscale images I are accumulated and generated from the event streams. L and I R , where the grayscale image I L represents the image generated by the left event camera; the grayscale image I R represents the image generated by the right event camera. Then, the 2D pixel coordinates of the center of the LED (calibration point) image in the image are extracted. and its 3D coordinates P(X W , Y W , 0) in the world coordinate system are associated; among them, the pixel coordinates are relative to the image coordinate system, the origin of the image coordinate system is the center of the image, the X-axis and Y-axis are parallel to the length and width of the image respectively, and the image coordinate system is a two-dimensional coordinate system; the world coordinate system is defined on the calibration board, the first calibration point or LED in the upper right corner of the calibration board 04 is the origin, the length and width directions of the calibration board 04 are the X-axis and Y-axis respectively; the Z-axis is perpendicular to the calibration board 04, see Figure 1 . Obtain the 2D pixel coordinates of the center of the image of a certain LED (calibration point) from the image coordinate system. The coordinates of this LED (calibration point) corresponding to the world coordinate system are P(X W , Y W , 0).

[0085] Force the camera coordinate system of the left event camera to completely coincide with the world coordinate system, then the external parameter matrix and offset of the left event camera are: R l = I, t l = 0; set the internal parameter matrix K l and the distortion coefficient D l of the left event camera; extract multiple groups of corresponding calibration points and their image coordinate points: P(X W , Y W , 0), In this embodiment, two coordinate points in a group correspond to an LED and the image of the LED in the image. Based on the monocular camera projection model:

[0086]

[0087] Construct a linear equation system to solve the camera projection matrix and obtain the initial value of the internal parameter K l . Among them: u l,i , v l,i represent the coordinates of the center of the i-th LED (calibration point) extracted in the image coordinate system of the event stream generated by the left event camera; X l,Wi , Y l,Wi represent the coordinates of the corresponding LED (calibration point) in the world coordinate system.

[0088] By minimizing the reprojection error objective function, K can be obtained. l and D l :

[0089]

[0090] where N is the number of calibration board postures used in the calibration process; M is the number of LED points extracted for each calibration board posture; is the 2D pixel coordinate of the center of the j-th marker point (LED) in the i-th posture observed by the left event camera; R l，ij is the theoretical 3D coordinate of the center of the j-th marker point (LED) in the i-th posture observed by the left event camera in the world coordinate system; p l，ij (K l , D l , P l，ij ) is the theoretical 2D coordinate after projecting the theoretical 3D coordinate P l，ij onto the image plane through the camera model.

[0091] Set the internal parameter matrix K r and the distortion coefficient D r of the calibrated right event camera, as well as the rotation matrix R1 and the translation vector t1 relative to the left event camera.

[0092] Use multiple groups of 3D-2D corresponding point coordinates of the right event camera, that is, in the image obtained by the right event camera, extract the world coordinates of multiple LEDs (calibration points) and the image coordinates of the corresponding images.

[0093] Based on the monocular camera projection model:

[0094]

[0095] Construct a linear equation system to solve the camera projection matrix and obtain the initial values of K r , R1, and t1. Among them, s represents the scale factor, u r,i , v r，i is the coordinate of the center of the image of an LED (calibration point) in the image generated by the right event camera, and X r，Wi , X r，Wi represent the coordinates of this LED (calibration point) in the world coordinate system.

[0096] Then, by jointly minimizing the binocular reprojection error objective function, K r , D r , R1, and t1 are obtained:

[0097]

[0098] where, The 2D pixel coordinates of the center of the j-th LED (calibration point) in the i-th pose observed by the right event camera; P r,ij The theoretical 3D coordinates of the center of the j-th LED (calibration point) in the i-th pose observed by the right event camera in the world coordinate system; p r,ij (K r , D r , R1, t1, P r,ij ) are the theoretical 2D coordinates after projecting the theoretical 3D coordinates P r，ij onto the image plane through the camera model.

[0099] When performing step 2, at least 4 marking points are set on the gun body 1 of the spray gun 01. Refer to Figure 2 . In this embodiment, 6 infrared LEDs 10 - 15 are set on the spray gun as marking points. Of course, other materials can also be used as marking points, such as reflective materials, fluorescent materials, high-contrast patterns, etc.; any 4 of the 6 infrared LEDs 10 - 15 are not coplanar; refer to Figure 1 . The spray gun 01 is installed on the flange at the operating end of the industrial robot 07, and the spray gun is controlled by the industrial robot 07; the industrial robot 07 is controlled by the industrial robot control system 06; the center point of the muzzle of the spray gun 01 touches the origin of the world coordinate system W, that is, the coordinate origin of the original calibration plate 04, through multiple different poses.

[0100] The left and right event cameras acquire event stream data for each calibration point. Specifically: on the premise that the 6 infrared LEDs 10 - 15 of the spray gun are within the fields of view of the dual event cameras 02 and 03, the center point of the muzzle of the spray gun 01 touches the origin of the world coordinate system W through 6 different poses, and the 6 infrared LEDs are made to flash to acquire event stream data.

[0101] The dual event cameras synchronously collect the event stream and send it to the industrial control computer 05, and the industrial control computer calculates the coordinates of the centers of the 6 infrared LEDs in the 6 poses in the world coordinate system W where i = 1, 2,..., 6 represents 6 infrared LEDs; n = 1, 2,..., 6 represents 6 different poses. The specific implementation includes:

[0102] Extract the two-dimensional coordinates of the infrared LED from the event stream (the two-dimensional coordinates in the image coordinate system);

[0103] Capture the event stream of the two event cameras for the n-th pose, and obtain the two-dimensional coordinates of the infrared LEDs that can be detected simultaneously in both the left and right event cameras and l represents the left camera event; r represents the right camera event; i = 1, 2,..., 6 represents 6 infrared LEDs.

[0104] 3D Reconstruction of Dual-Event Camera

[0105] Using the calibration parameters of the dual-event camera, for each marked point, construct the projection equation and solve it by the least squares method to convert the coordinates of the matched 2D points into 3D coordinates

[0106]

[0107] s l ,s r are the scale factors of the left and right cameras respectively.

[0108] Calculate the rigid offset ΔP i :

[0109] In the nth pose, assume the rigid transformation of the spray gun is R n , and the translation vector t n =0, minimize the error objective function of the observed coordinates and the model-predicted coordinates in all poses:

[0110]

[0111] Then the rigid offset ΔP from the i-th infrared LED to the center point of the gun muzzle can be obtained i .

[0112] When performing step 3: Calibrate the coordinate system T of the center point of the industrial robot spray gun muzzle;

[0113] Specifically include: Set the direction of the tool coordinate system T of the center point of the spray gun muzzle to be consistent with the flange center coordinate system F. The industrial robot controls the center point of the spray gun muzzle to touch the origin of the world coordinate system W in different poses, and the industrial robot control system records the poses T of the flange center in the base coordinate system B in 4 different poses respectively F , and solve the homogeneous transformation matrix by the least squares method to obtain the rigid offset of the center point of the spray gun muzzle relative to the flange center; then the homogeneous transformation matrix from the tool coordinate system T to the industrial robot base coordinate system B can be obtained where, is the homogeneous transformation matrix from the industrial robot flange center coordinate system F to the base coordinate system B; that is, establish the tool coordinate system T of the center point of the industrial robot spray gun muzzle.

[0114] Establish the homogeneous transformation matrix from the world coordinate system W to the industrial robot base coordinate system B which is directly obtained by using the industrial robot to control the center point of the spray gun muzzle to calibrate the user coordinate system for the calibration plate coordinate system (world coordinate system W).

[0115] Specifically, the industrial robot controls the movement of the spray gun so that the center of the spray gun nozzle touches the origin of the world coordinate system defined on the calibration plate to obtain the origin data; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the X-axis side of the calibration plate to obtain the movement data information in the positive X-axis direction; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the Y-axis side of the calibration plate to obtain the movement data information in the positive Y-axis direction; thus, the homogeneous transformation matrix can be calculated.

[0116] Preferably, to improve the control accuracy, time synchronization adjustment is required:

[0117] The Precision Time Protocol (PTP) is used to synchronize the time of the teaching system of the spraying robot, including:

[0118] Based on the IEEE1588 protocol, the industrial control computer is used as the master clock to periodically send Sync Messages, and the spray gun controller, dual-event camera, and industrial robot control system are used as slave clocks to receive and record timestamps. The time offset is:

[0119]

[0120] where Δt delay is the transmission delay.

[0121] Then the spray gun controller, two event cameras, and industrial robot control system adjust their local clocks according to the calculated time offset, and the adjusted clock time is:

[0122] t’ Gun = t Gun + Δt offset

[0123] where t Gun is the local time before adjustment.

[0124] When performing the teaching in step 4:

[0125] Refer to Figure 1 、 Figure 2, the spray gun 01 is removed from the industrial robot 07, and the spray gun 01 is held by hand for spray teaching; two event cameras collect event streams in real time during the teaching process, and at the same time, the pressure sensor 6 in the spray gun 01 collects pressure data streams, which are used to indirectly collect the size of the ejected paint volume of the spray gun 01. When specifically setting, a pressure sensor 6 is installed at the pressing place of the trigger 5 inside the spray gun handle, and the pressure data when the trigger 5 is depressed can be obtained in real time; a servo motor 3 is installed inside the gun body, and is connected to the needle 2 of the spray gun 01 through a connector 4, which can drive the needle 2 to move back and forth, adjust the gap size between the needle 2 and the front nozzle of the gun body 1, and thus control the size of the paint ejection volume; a controller 4 is installed inside the gun body 1, which calculates the displacement of the needle 2 in real time according to the pressure data obtained by the pressure sensor 6, that is, the controller inside the gun body controls the size of the paint ejection volume of the nozzle jet according to the pressure data, and then sends a signal to the servo motor 3; a communication module 8 is installed inside the gun body 1, which is used to send the displacement data of the needle 2 during the teaching process to the industrial control computer. The industrial control computer is equivalent to a computing and processing center and is used to process data.

[0126] When reproducing after teaching, in addition to trajectory reproduction, the robot 07 also needs to reproduce the size of the ejected paint volume of the spray gun 01. The size of the ejected paint volume is related to pressure and needle position. Therefore, the trigger can be directly controlled according to the pressure information to control the size of the paint ejection volume of the spray gun during operation, or the moving distance of the trigger can be controlled according to the needle position to achieve the purpose of controlling the size of the paint ejection volume of the spray gun during operation.

[0127] Secondly, when the two event cameras are collecting, the marking points outside the spray gun are infrared LEDs. As Figure 2 shown, the six infrared LEDs can be set separately. Two are set above the front end of the spray gun, one is set below the middle part, one is set on each side of the rear end, and one is set above the rear end. Of course, other positions can also be set.

[0128] During shooting, events are generated only when the brightness change of the infrared LED exceeds the threshold, and some LEDs may be invisible due to occlusion. Therefore, the generated event stream is an asynchronous and sparse spatio-temporal data set:

[0129] E = {e k}

[0130] where each event e k = (x k , y k , t k , p k ), where: x k , y k are pixel coordinates, t k is the timestamp, and p k is the polarity;

[0131] The controller 7 of the spray gun 01 calculates the displacement of the needle in real time according to the pressure data obtained by the pressure sensor. Sends a signal to the servo motor to adjust the gap between the needle and the front nozzle of the gun body, and synchronously records the displacement of the needle. Sends the needle displacement data or pressure data with a timestamp to the industrial control computer through the communication module.

[0132] After the teaching is completed, press the on / off button on the handle to switch the spray gun from the teaching state to the closed state.

[0133] When performing step 5: Data processing:

[0134] The industrial control computer receives the event streams of the two event cameras and the needle displacement data sent by the spray gun controller. Perform data processing:

[0135] Align the time of the event streams of the two event cameras and segment the event streams:

[0136] A change in the movement speed of the spray gun will cause the density of the event stream to be uneven. The event stream can be segmented according to the number of events. For example, set a threshold of every 10,000 events as a segment. When the number of events of both the left and right cameras reaches the threshold, the event streams of the left and right cameras within this time period are respectively divided into a segment.

[0137] Obtain the pose of the center point of the spray gun nozzle in the world coordinate system by calculating each segment of the event stream corresponding to the left and right cameras. It includes:

[0138] Preprocess the event stream:

[0139] First, perform time window filtering: Using the earlier event timestamp of the two event cameras as the starting point of the window, set the time window Δt = 1ms, and for each event e i in each segment of the event stream E k , count the number of events in its spatial neighborhood (3×3 pixels) within the time window [t k -△t,t k +△t]. If the number is greater than or equal to 5, then retain the event, and finally obtain the new event stream E f ;

[0140] Then divide the event stream E f into 6 candidate clusters of infrared LEDs:

[0141] Randomly select 6 events e1, e2, e3, e4, e5, e6 as the initial clustering centers. For each data point e f in the event stream E i , calculate its distance from the 6 clustering centers. According to the distance value, classify e iAllocate it to the cluster to which the nearest clustering center belongs.

[0142] For each formed cluster, calculate the mean of all events within the cluster, and take the e closest to the mean i As the new clustering center, and then repeat the three steps of calculating distance values, allocating clusters, and updating the clustering center until the clustering center no longer changes.

[0143] Dual-camera 3D reconstruction:

[0144] Perform weighted averaging on the n event coordinates within each LED cluster:

[0145]

[0146] Among them, Is the weighted value of the event in the left camera event stream, and d i Is the average distance from this event to other events within the cluster.

[0147] Calculate the two-dimensional coordinates of the same infrared LED in the left and right event cameras And Then, using the calibration parameters of the dual-event cameras, solve by the least squares method to convert the two-dimensional coordinate pairs (image coordinate system) of the matching points into three-dimensional coordinates (World coordinate system):

[0148]

[0149] Calculate the pose of the spray gun:

[0150] Minimize the error objective function of the observed coordinates of the calibrated points (LEDs) obtained above and the model predicted coordinates:

[0151]

[0152] Then R and t can be obtained, where: the translation vector t is the current position value of the center point of the spray gun nozzle, and the rotation matrix R is the current pose of the spray gun in the world coordinate system.

[0153] Convert the pose of the center point of the spray gun nozzle in the world coordinate system To the pose of the center point of the nozzle in the base coordinate system of the industrial robot:

[0154]

[0155] Finally, align the pose data and the needle displacement data according to the PTP timestamp difference.

[0156] Perform Step 6:

[0157] Generate the program:

[0158] The industrial control computer generates a control program for controlling an industrial robot to perform a spraying operation and sends it to the industrial robot control system. By reinstalling the spray gun described in claim 1 at the end of the industrial robot, the program can be run for reproduction.

[0159] The program includes industrial robot motion trajectory control instructions and spray gun servo motor control instructions to achieve synchronous and precise control of the spray gun pose and the gradually changing pressing force of the spray gun.

[0160] The above content is only a preferred embodiment of the present invention. For those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. The content of this specification should not be construed as a limitation of the present invention.

Claims

1. A teaching method for the spray gun of a spraying robot based on a dual-event camera, characterized in that: It includes the following steps: Step 1: Calibrate the internal and external parameters of two event cameras in combination with a calibration board; Step 2: The spray gun is set with at least 4 marking points to obtain the rigid offset from the marking points to the center point of the gun muzzle; Step 3, obtain the homogeneous transformation matrix from the tool coordinate system T of the muzzle center to the base coordinate system B of the industrial robot and the homogeneous transformation matrix from the world coordinate system W to the base coordinate system B of the industrial robot Step 4: Teaching. Obtain the event streams of two event cameras for the marking points on the spray gun, and the time stream of the paint discharge volume of the spray gun during spraying; Step 5: According to the event stream and the rigid offset, obtain the pose of the center point of the spray gun muzzle in the world coordinate system W, and convert it to the pose of the center point of the gun muzzle in the base coordinate system B of the industrial robot, and obtain the time stream of the pose of the center point of the gun muzzle; Step 6: Generate a control program for the industrial robot according to the time stream of the paint discharge volume during spraying and the time stream of the pose of the center point of the gun muzzle.

2. The method for teaching the spray gun of the spraying robot based on the dual-event camera according to claim 1, wherein: The dual-event cameras are the left event camera and the right event camera respectively. When performing step 1, the left event camera and the right event camera synchronously acquire event streams for the calibration board, and respectively generate a series of grayscale images I L and I R ; Set the world coordinate system W and the image coordinate system. Among them, the world coordinate system W is established on the calibration board. Obtain the coordinates of the calibration points on multiple calibration boards and the coordinates of the calibration point images in the image, and then calculate the internal parameters and external parameters of the two event cameras respectively through camera coordinate system conversion.

3. The method for teaching the spray gun of a spraying robot based on a dual-event camera according to claim 2, characterized in that: The calibration board is an array of LED boards, with multiple LEDs distributed. The LEDs are calibration points, and the LEDs flash according to a set program. The left and right event cameras synchronously collect event streams of the calibration board; different poses of the calibration board corresponding to the images are selected, and the 2D pixel coordinates of the centers of the LED images in the images are extracted and its 3D coordinates P(X W , Y W , 0) in the world coordinate system.

4. The method for teaching the spray gun of the spraying robot based on the dual-event camera according to claim 2 or 3, characterized in that: By calculation, the internal parameters and external parameters of the two event cameras are obtained respectively, including: Force the camera coordinate system of the left event camera to coincide exactly with the world coordinate system. Then, the external parameter matrix and offset of the left event camera are: R l = I, t l = 0; Set the internal parameter matrix K l and distortion coefficient D l ; Extract multiple sets of corresponding coordinate points: P(X W , Y W , 0), Two coordinate points in a group correspond to a calibration point and the image of the calibration point in the image; Based on the monocular camera projection model: Construct a system of linear equations to solve the camera projection matrix and obtain the initial value of the internal parameter K l . Among them: u l，i , v l，i represent the coordinates of the center of the i-th LED extracted in the image coordinate system; X l，Wi , Y l，Wi represent the coordinates of the calibration point in the world coordinate system; Then, the combined minimization of the binocular reprojection error objective function is adopted to obtain K l and D l : where N is the number of calibration board postures used in the calibration process; M is the number of LED points extracted for each calibration board posture; is the 2D pixel coordinate of the center of the j-th LED in the i-th posture observed by the right event camera; P r，ij is the theoretical 3D coordinate of the center of the j-th LED in the i-th posture observed by the right event camera in the world coordinate system; P r，ij (K r , D r , R1, t1, P r，ij ) is the theoretical 2D coordinate after projecting the theoretical 3D coordinate P r，ij onto the image plane through the camera model; Calibrate the intrinsic matrix K of the right event camera r and the distortion coefficient D r , as well as the rotation matrix R1 and the translation vector t1 relative to the left event camera. Use multiple groups of 3D-2D corresponding point coordinates of the right event camera, that is, in the image obtained by the right event camera, extract the world coordinates of multiple calibration points and the image coordinates where the corresponding images are located. Based on the monocular camera projection model: Construct a system of linear equations to solve for the camera projection matrix and obtain the initial values of K r , R1, and t1. Here, s is the scale factor, and u r，i , v r，i are the coordinates of the image center of a certain calibration point in the image generated by the right-event camera, and X r，Wi , X r，Wi represent the coordinates of this calibration point in the world coordinate system. Then, by jointly minimizing the binocular reprojection error objective function, K r , D r , R1, and t1 are obtained: Among them, is the 2D pixel coordinate of the center of the j-th calibration point in the i-th pose observed by the right event camera; P r，ij is the 3D coordinate of the center of the j-th calibration point in the i-th pose observed by the right event camera in the world coordinate system; p r，ij (K r , D r , R1, t1, P r，ij ) is the 2D coordinate after projecting the theoretical 3D coordinate P r，ij onto the image plane through the camera model.

5. The spray gun teaching method of the spraying robot based on the dual-event camera according to claim 1, wherein: When performing step 2, there are L marking points (L≥4) set on the spray gun. The center point of the spray gun nozzle touches the origin of the world coordinate system W in multiple different postures, and the left and right event cameras acquire event streams for each marking point; according to the event stream data of the marking points, the coordinates of the L marking points in the world coordinate system W under K postures are extracted; and the two-dimensional coordinates of the corresponding marking points in the image coordinate system are obtained; both K and L are natural numbers; the two-dimensional coordinates of the marking points detected in the left and right event cameras are obtained and n represents the nth posture, n = 1, 2, 3... K; l represents the left image generated by the left event camera, r represents the right image generated by the right view camera, and i represents the ith marking point, i = 1, 2, 3... L; According to the internal and external parameters of the double-event camera, for each marked point, a projection equation is constructed and solved by the least squares method to convert the coordinates of the matched two-dimensional points into three-dimensional coordinates Calculate the rigid offset ΔP i : In the nth posture, let the rigid transformation of the spray gun be R n , and the translation vector t n = 0; Adopt the error objective function that minimizes the observed coordinates and the model predicted coordinates under all poses: The rigid offset ΔP from the i-th marked point to the center point of the muzzle can be obtained i .

6. The method for teaching the spray gun of a spraying robot based on a dual-event camera according to claim 1, wherein: When performing Step 3: Set the T direction of the tool coordinate system at the center point of the spray gun nozzle to be consistent with the F coordinate system at the flange center. The industrial robot controls the center point of the spray gun nozzle to touch the origin of the world coordinate system W in different postures, and the industrial robot control system records the poses of the flange center in the base coordinate system B in 4 different postures respectively T F , and solve the homogeneous transformation matrix by the least squares method Then the homogeneous transformation matrix from the tool coordinate system T to the base coordinate system B of the industrial robot can be obtained. Among them, is the homogeneous transformation matrix from the flange center coordinate system F of the industrial robot to the base coordinate system B. Establish the homogeneous transformation matrix from the world coordinate system W to the base coordinate system B of the industrial robot It is directly obtained by calibrating the user coordinate system of the industrial robot to the calibration plate coordinate system (world coordinate system W) by using the center point of the spray gun nozzle Specifically, the industrial robot controls the movement of the spray gun so that the center of the spray gun nozzle touches the origin of the world coordinate system defined on the calibration plate to obtain the origin data; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the X-axis side of the calibration plate to obtain the movement data information in the positive X-axis direction; then controls the center of the spray gun nozzle to move from the origin of the world coordinate system to a point on the Y-axis side of the calibration plate to obtain the movement data information in the positive Y-axis direction; thus, the homogeneous transformation matrix can be calculated 7. The method for teaching the spray gun of a spraying robot based on a dual-event camera according to claim 1, wherein: When performing Step 4 for teaching: Remove the spray gun from the industrial robot and hold the spray gun for spraying teaching; Two event cameras collect event streams in real time during the teaching process, and at the same time, a pressure sensor in the spray gun collects pressure data; The controller in the gun body controls the paint discharge volume of the nozzle according to the pressure data and obtains the time stream of the paint discharge volume of the nozzle.

8. The method for teaching the spray gun of a spraying robot based on a double-event camera according to claim 1, characterized in that: When performing Step 5: Receive the event streams of the two event cameras and the data of the paint discharge volume of the nozzle controlled by the spray gun controller, and perform data processing: align the time of the event streams of the two event cameras and segment the event streams; Obtain the pose of the center point of the spray gun muzzle in the world coordinate system by calculating each segment of the event stream corresponding to the left and right cameras, including: preprocessing the event stream: first performing time window filtering: using the earlier-arriving event timestamp in the two event cameras as the window start point, setting the time window Δt = 1ms, and for each event e i in each segment of the event stream E k , count the number of events in its spatial neighborhood within the time window [t k - Δt, t k + Δt]. If the number is greater than or equal to 5, retain the event. Finally, obtain the new event stream E f ; Then, the event stream E f is segmented into candidate clusters of L marker points: Randomly select L events e1, e2,......, e L as the initial clustering centers. For the event stream E f for each data point e i in it, calculate its distances from the L clustering centers. According to the distance values, assign e i to the cluster to which the nearest clustering center belongs. For each formed cluster, calculate the mean of all events within the cluster, and take the e closest to the mean i as the new clustering center, and then repeat the three steps of recalculating the distance values, assigning to clusters, and updating the clustering center until the clustering center no longer changes.

9. The spray gun teaching method of the spraying robot based on the dual-event camera according to claim 8, wherein: It also includes: Perform weighted average on the n event coordinates in each candidate cluster of marking points: Among them, is the weighted value of the event in the left camera event stream, and d i is the average distance from this event to other events within the cluster; Calculate the two-dimensional coordinates of the same marked point in the left and right event cameras and Then, using the calibration parameters of the dual-event camera, solve by the least squares method to convert the two-dimensional coordinate pairs of the matching points into three-dimensional coordinates Calculate the pose of the spray gun: Minimize the error objective function of the observed coordinates and the model predicted coordinates of the marking points obtained above: Then R and t can be obtained, where: the translation vector t is the current position value of the center point of the spray gun muzzle, and the rotation matrix R is the current pose of the spray gun in the world coordinate system; The pose of the center point of the spray gun nozzle in the world coordinate system is converted into the pose of the center point of the nozzle in the base coordinate system of the industrial robot: Obtain the time stream of the pose of the center point of the nozzle.

Images