A water cutting collaborative dual-robot arm safety protection method based on machine vision

Abstract

The application discloses a water cutting cooperative double-robot arm safety protection method based on machine vision, which comprises the following steps: step 1, establishing a robot arm kinematics model; step 2, acquiring robot arm key point coordinates; step 3, acquiring a target image, calibrating a camera, and shooting the target image; step 4, processing the image to acquire a water jet outer contour image; step 5, acquiring target point P2 coordinates; step 6, calculating the shortest distance between a water jet axis and a robot arm; and step 7, contact risk assessment. The application has the beneficial effect that high-precision image acquisition and processing technology of machine vision can realize high-precision calculation of the spatial position of a divergent water jet axis point. The machine vision system can acquire and process images in real time, realize real-time calculation of the spatial position of the divergent water jet axis point, and be suitable for a dynamic divergent water jet scene.

Description

Technical Field

[0001] This invention relates to a safety protection method for dual robotic arms, and more particularly to a safety protection method for dual robotic arms using water cutting based on machine vision, belonging to the field of robotic arm collision detection technology. Background Technology

[0002] Waterjet cutting dual-arm collaborative robots are crucial equipment for automated cutting operations. Real-time collision detection between the two arms forms a core element of their technical architecture, enabling them to perform automated cutting tasks and monitor operational status. Specifically, during operation, clamping devices at the ends of the arms stabilize the object being cut, while the main arm is equipped with a water jet to perform precise waterjet cutting. It is noteworthy that during waterjet cutting, the high-speed, divergent water jet approaches the robotic arm; however, current technology lacks a mature method to accurately detect the spatial pose of this divergent water jet, thereby preventing equipment damage and ensuring the continuity and stability of waterjet cutting operations.

[0003] Chinese patent CN109773785A discloses a collision avoidance method for industrial robots. First, the geometric models of the robot and workpiece are updated based on the predicted robot motion state. Intersection detection of the robot and workpiece geometric models is then performed. After confirming there is no collision risk, an AABB bounding box model is established based on the updated robot and workpiece geometric models, and intersection detection is performed with the simplified workpiece AABB bounding box model. Finally, intersection detection is performed between the robot, workpiece, and the external and internal tooling within the intersection area. This method requires testing with multiple models. While it can prevent collisions, the geometric models of the workpiece and robot are known, making collision detection impossible for changing objects.

[0004] Chinese patent CN103192413A discloses a sensorless robot collision detection device and method. The method involves a computing module calculating the difference between the robot's predicted motion state at the next moment and its actual planned state, based on the prediction from an observation module. A judgment module then compares this difference to a preset threshold. If the difference exceeds the threshold, the execution module stops the robot. However, this method requires a large number of samples and significant time investment for neural network training, making it unsuitable for industrial robots in flexible manufacturing systems that require frequent tool changes. Furthermore, this method only stops the robot when a collision occurs, failing to prevent collisions from happening in the first place.

[0005] Chinese patent CN116604557A discloses a collaborative collision avoidance planning optimization method for dual-arm robots in unstructured environments. The method uses a depth camera to acquire three-dimensional point cloud information of the surrounding environment, establishes a safety envelope for collaborative robot operation in a constrained environment based on the swept sphere method, obtains the distance between the safety envelopes using computational graphics algorithms, and establishes mathematical models and dynamic programming methods according to different tasks and objectives to obtain the optimal trajectory.

[0006] Existing collision detection methods mainly fall into two categories: offline simulation based on path planning and real-time detection based on torque feedback. Offline simulation requires pre-written programs, cannot handle different working conditions, and is difficult to modify the model for real-time conditions. Real-time detection methods based on torque feedback mainly aim to detect collisions and stop them in time by analyzing abnormal torque fluctuations at the time of a collision. However, these methods cannot actually prevent collisions from occurring and often lead to false alarms due to excessively high detection sensitivity.

[0007] In practical applications, water jets often exhibit non-uniform divergent patterns due to environmental factors and physical properties. Traditional machine vision technology mainly focuses on the detection of objects with fixed geometric features, and its positioning accuracy is often difficult to guarantee when faced with dynamic and varied divergent water jets. Summary of the Invention

[0008] Purpose of the Invention: The purpose of this invention is to provide a safety protection method for collaborative dual-arm water jet cutting based on machine vision. During collaborative dual-arm operation, the target is fixed by a clamping device at the end of the secondary arm, while the primary arm uses a water jet to emit a high-pressure water jet to cut the object. This invention utilizes machine vision technology to achieve real-time spatial positioning of key points along the axis of the divergent water jet released by the primary arm's water jet, thereby obtaining its spatial pose information. This method prevents the divergent water jet emitted by the primary arm's water jet from contacting the clamping device of the secondary arm during collaborative dual-arm operations, thus avoiding damage to the equipment and improving the safety and efficiency of the system operation.

[0009] Technical solution: A safety protection method for water jet cutting collaborative dual robotic arms based on machine vision, comprising the following steps:

[0010] Step 1: Establish the kinematic model of the robotic arm, construct a fixed global world coordinate system and the local coordinate system of each link of the robotic arm, and perform kinematic modeling of the two robotic arms working in cooperation;

[0011] Step 2: Obtain the coordinates of key points of the robotic arm, transform the coordinates to associate the local coordinate system of each link of the robotic arm that was established in Step 1 with the global world coordinate system, and obtain the three-dimensional coordinate information of key point P1 on the end effector of the main robotic arm and point M from the center point of the end effector of the robotic arm in the global world coordinate system.

[0012] Step 3: Acquire the target image, calibrate the camera, and capture the target image;

[0013] Step 4: Process the image to obtain the outer contour image of the water jet;

[0014] Step 5: Obtain the coordinates of target point P2, extract target point P2 on the water jet axis, and calculate and determine the three-dimensional spatial position of target point P2 in the global world coordinate system.

[0015] Step 6: Calculate the shortest distance between the water jet axis and the robotic arm. Based on the points P1 and P2 obtained in Step 2 and Step 5, calculate the mathematical equation of the straight line L passing through P1 and P2 in the global world coordinate system, and calculate the shortest distance d between the straight line L and the water jet axis.

[0016] Step 7: Risk assessment. Establish a safe distance threshold r. By comparing the shortest distance d with the safe distance threshold r, if the shortest distance d is greater than the safe distance threshold r, return to step 3. If the shortest distance d is less than or equal to the safe distance threshold r, it is determined that there is a risk of contact and protective measures are taken.

[0017] In real-world operating environments, a water jet is ideally considered as a regular cylinder. However, when a robot performs a cutting task, the water jet exhibits various divergent phenomena, with its axial vector position and length dynamically changing. Faced with this complex and ever-changing scenario, traditional positioning methods are inadequate in capturing and locating the position and direction of divergent water jets. This invention integrates kinematic principles and machine vision algorithms to determine the spatial pose of divergent water jets. Particularly for real-time changing and unpredictable divergent water jet patterns, this technology can quickly and accurately capture their dynamic characteristics. By locating divergent water jets in real time, it can determine whether the device is in a dangerous state, thereby preventing high-speed jet cutting of the robotic arm and achieving safe and reliable operation.

[0018] Preferably, in order to accurately derive the three-dimensional spatial coordinates of a specific target point on the end effector of the dual robotic arms, the method for establishing the kinematic model of the robotic arms in step 1 is as follows:

[0019] 1.1 Establishing the coordinate system of the dual-arm linkage using the DH method

[0020] a i = Along X i Axis, from Zi Move to Z i+1 distance,

[0021] α i =around X i Axis, from Z i Rotate to Z i+1 Angle,

[0022] d i = Along Z i Axis, from X i-1 Move to X i distance,

[0023] θ i = Around Z i Axis, from X i-1 Rotate to X i Angle;

[0024] 1.2 Establish the link coordinate system of the robotic arm based on the DH parameters and the dimensions of the robotic arm.

[0025] By constructing a fixed global world coordinate system and the local coordinate system of each link of the robotic arm, kinematic modeling of the two robotic arms working in cooperation is performed. The Denavit-Hartenberg (DH) parameter method is used for kinematic analysis to accurately derive the three-dimensional spatial position coordinates of a specific target point on the end effector of the two robotic arms.

[0026] The preferred method for obtaining the coordinates of the key points of the robotic arm in step 2 is as follows:

[0027] 2.1. Based on the robotic arm linkage coordinate system established in step 1, the key points P1(X1,Y1,Z1) and M(X) are determined through coordinate transformation. m ,Y m Z m The coordinate position in the world coordinate system.

[0028] The key point P1 is located at the intersection of the water gun's axis and the axis of the last section of the robotic arm.

[0029] The key point M is located at the center of the spherical enclosure of the clamping device;

[0030] 2.2. Obtain the forward kinematic equations of the robotic arm, and then obtain the transformation matrix between the links using matrix multiplication. i T j Let {j} represent the transformation matrix relative to coordinate system {i}. The overall transformation matrix of the robotic arm's terminal control mechanism relative to the base is expressed as:

[0031] 0 T6 = 0 T11 T2 2 T3 3 T4 4 T5 5 T6 (1)

[0032] Based on the established relationship between coordinate systems, coordinate transformation technology is used to obtain the three-dimensional coordinate information of the key point P1 on the end effector of the main robot arm and the center point M of the end effector of the robot arm in the global world coordinate system.

[0033] Preferably, in order to achieve camera positioning, the method for calibrating the camera in step 3 is as follows:

[0034] Based on the principle of pinhole imaging, the relationship between the image in the pixel coordinate system, the image coordinate system, and the camera coordinate system is determined. The transformation relationship between the three coordinate systems is described as shown in formula (2):

[0035]

[0036] Suppose p is a point in space, and its coordinates in the world coordinate system are... Its projection onto the pixel coordinate system has coordinates (u, v); This is the intrinsic parameter matrix of the camera. The extrinsic parameter matrix of the camera is obtained through camera calibration. The intrinsic parameter matrix and the extrinsic parameter matrix are collectively referred to as the camera matrix.

[0037] The camera calibration method mentioned is Zhang Zhengyou's camera calibration method, and the specific method is as follows:

[0038] S1. Prepare a chessboard pattern and take pictures of it from different angles with a camera to obtain a set of images;

[0039] S2. Detect the corner points of the calibration board in the image, obtain the pixel coordinate values ​​of the corner points of the calibration board, and then calculate the physical coordinate values ​​of the corner points of the calibration board.

[0040] S3. Solve for the intrinsic and extrinsic parameter matrices;

[0041] S4. Optimize the above parameters using the LM (Levenberg-Marquardt) algorithm.

[0042] Camera calibration is the prerequisite and foundation for the successful application of machine vision in practice. Only by accurately calibrating the camera can the effective three-dimensional position information of the object be accurately calculated based on the acquired two-dimensional image sequence of the object.

[0043] Preferably, in order for the machine vision system to acquire and process images in real time, and to realize the real-time calculation of the spatial position of the axis point of the divergent water jet, applicable to dynamically changing divergent water jet scenarios, the image processing method in step 4 is as follows:

[0044] 4.1. Through image binarization processing, extract the outer contour information of the water jet from the water cutting robot arm operation image captured by the camera; set a threshold T to separate the two types of pixels. Any point (x,y) in the image with f(x,y)>T is the target point, otherwise the point is the background point. Distinguish between target pixels and background pixels, remove messy background information, and at the same time retain the image features of the high-speed water jet.

[0045] 4.2 Edge detection: Perform edge detection on the binarized image to obtain the required outer contour of the water jet and remove pixels that are not edge-defined.

[0046] From the images of the water jet cutting robot captured by the camera, given that the high-pressure water jet appears as a bright white area due to light scattering, its grayscale value is usually low. In order to accurately extract the outer contour information of the water jet, effectively remove cluttered background information, and preserve the image features of the high-speed water jet as completely as possible, we adopt an image binarization processing strategy.

[0047] Due to the influence of background and lighting, image binarization alone cannot completely separate the water jet contour from the background. Therefore, edge detection is required on the binarized image to obtain the desired water jet contour and remove some non-edge pixels.

[0048] The edge detection method is the Canny edge detection algorithm, and the specific method is as follows:

[0049] Step 1: First, perform Gaussian filtering on the input image to remove noise;

[0050] Step 2: On the smoothed grayscale image, use the Sobel operator to calculate the gradient magnitudes Gx(x,y) and Gy(x,y) in the horizontal and vertical directions;

[0051] Step 3: Non-maximum suppression. After calculating the gradient magnitude and direction, remove non-edge points.

[0052] Step 4: Use the Otsu algorithm to implement the adaptive selection of high and low thresholds in the Canny algorithm.

[0053] Preferably, in order to accurately obtain the coordinates of the target point P2, the method for obtaining the coordinates of the target point P2 in step 5 is as follows:

[0054] 5.1 Extracting straight lines: Based on the characteristic that the water jet contour is a straight line, straight line detection is performed on the image to obtain the complete outer contour of the water jet.

[0055] 5.2 Extract the four corner points of the target to obtain the outer contour of the water jet. Then extract the four corner points of the outer contour to calculate the position of a point P on the water jet axis in the image coordinate system.

[0056] 5.3 Obtain the three-dimensional coordinates of the target point P2. The object of image processing is a divergent water jet, which is similar to a frustum in space. The corner points obtained after image processing are regarded as the four vertices of a convex quadrilateral. The center position of the four points is on the axis. Calculate the center position of these four points to obtain a point on the axis.

[0057] Using the principle of binocular stereo vision positioning and coordinate transformation, the three-dimensional coordinates P2(X2,Y2,Z2) of the center point p in the world coordinate system are obtained. The desired P2 is a point on the axis of the divergent water jet.

[0058] Preferably, in order to obtain the complete outer contour of the water jet, the method for extracting straight lines is Hough transform extraction, and the specific method is as follows:

[0059] Step 1: Obtain all points with a value of 1 in the binary image as target points. Perform a Hough transform on each target point to map it to the parameter space, and quantize the P and θ parameters into M. P and N θ Equal portions;

[0060] The second step is to determine the discretized M... P and N θ Create a two-dimensional accumulator A with an initial value of 0;

[0061] The third step is to calculate the coordinates of each point in the Cartesian coordinate system in polar coordinates. For each additional sinusoidal intersecting curve generated, increment the accumulator by 1.

[0062] The fourth step is to select an appropriate threshold for judgment. When the value in the judgment accumulator is greater than the threshold, the matched line is the required straight line. Points on the same line appear as peak points in polar coordinates. Then, the length of the line is calculated and returned to Cartesian coordinates to complete the extraction of the straight line.

[0063] After obtaining the edges of the image, which are not complete straight lines, we can perform straight line detection on the image based on the characteristic that the water jet contour is a straight line, in order to obtain the complete outer contour of the water jet.

[0064] Preferably, to accurately determine the positions of the four corner points of the target, the method for extracting the four corner points of the target is Harris corner detection, and the specific method is as follows:

[0065] After obtaining the outer contour line of the water jet, it is necessary to extract the four corner points of the outer contour to calculate the position of a point P on the axis of the water jet in the image coordinate system.

[0066] The high-pressure water jet ejected by the water gun is a cylinder under ideal conditions. In practice, it will diverge, similar to a frustum of a cone. In either case, when performing corner detection on the image, the center point of the four corner points finally detected, that is, the centroid of the quadrilateral, and the center point is on the axis of the water jet, all meet the purpose of the detection in this step. The center points of corner points (1, 2, 3, 4) and corner points (1, 2, 5, 6) are the same point and on the central axis of the water jet.

[0067] The gray-scale transformation generated by the movement of the local window on the image is used to determine whether it is a corner point by the degree of change of the window in each direction; the image window (u, v) is translated to generate the gray-scale change E(u, v).

[0068] Obtain the image coordinates of the four corner points, obtain the centroid coordinates of the water jet image in the image coordinate system, and use formula (2) to obtain the coordinates P2 of P in the global world coordinate system.

[0069] E(u, v) = ∑ x,y w(x, y)[I(x + u, y + v) - I(x, y)] 2 (3)

[0070] The above formula is written as

[0071]

[0072] In the formula,

[0073]

[0074] The w function is the window function, and the M matrix is the partial derivative matrix. Among them, I x 、I y are the partial derivative functions of the image in the horizontal and vertical directions respectively. According to the eigenvalue calculation method, 2 eigenvalues are generated. These 2 eigenvalues represent the intensity of the transformation of each pixel point. In the actual calculation process, the corner response function is used to calculate the response value R of each corner point, and then whether it is a corner point is judged according to the R value;

[0075] If the set threshold Tr < R, then it is considered that this pixel point is a corner point in the image, otherwise it is a non-corner point in the image;

[0076] The mathematical definition of the corner response function is specifically as follows:

[0077] R = det(M) - k(traceM) 2 (6)

[0078] In the formula, k is an empirical constant, and the value of k ranges from 0.04 to 0.06.

[0079] Preferably, to accurately obtain the three-dimensional coordinates of the target point P2, the method for obtaining the three-dimensional coordinates of the target point P2 is as follows:

[0080] Because the object of image processing is a diverging water jet, which resembles a frustum in space, the corner points obtained after image processing are regarded as the four vertices A of a convex quadrilateral. i (u i ,v i The centers of the four points lie on the axis. Calculating the center of these four points will find a point on the axis.

[0081]

[0082] Using the principle of binocular stereo vision positioning and coordinate transformation, the three-dimensional coordinates P2(X2,Y2,Z2) of the center point p in the world coordinate system are obtained. The desired P2 is a point on the axis of the divergent water jet.

[0083] The preferred method for calculating the shortest distance between the water jet axis and the robotic arm in step 6 is as follows:

[0084] Based on the obtained P1(X1,Y1,Z1) and P2(X2,Y2,Z2), a spatial straight line L is obtained that is parallel to the axis of the divergent water jet.

[0085] The two-point form of the equation of line L:

[0086]

[0087] The M(X) obtained from step 1 m ,Y m Z m ), calculate the shortest distance d between M and L;

[0088] The value of d is calculated from the distance from the point to the line. Taking a point P1 on the line, we obtain...

[0089]

[0090] Direction vector of line L

[0091]

[0092] but

[0093]

[0094] Where, a=(Y1-Ym (Z2-Z1)-(Y2-Y1)(Z1-Z m b = (X1 - X) m (Z2-Z1)-(X2-X1)(Z1-Z m ), c = (X1 - X m (Y2-Y1)-(X2-X1)(Y1-Y m );

[0095] Find the shortest distance d between M and L:

[0096]

[0097] Beneficial effects: Kinematic modeling and analysis of the robotic arm in this invention: Kinematic analysis is performed using the Denavit-Hartenberg (DH) parameter method to accurately derive the three-dimensional spatial coordinates of a specific target point on the end effector of the dual robotic arms.

[0098] Image acquisition system deployment: Utilizing high-resolution machine vision sensors, continuous and high-precision image capture of the water jet is achieved. The vision sensor is mounted between the two robotic arms to ensure clear recording of the water jet's diffusion pattern and key characteristics.

[0099] Image preprocessing and feature extraction: The acquired image data is preprocessed, including noise suppression and image binarization, to improve image quality. Subsequently, advanced image processing techniques such as edge detection, line fitting, and corner detection are used to extract the contour boundary and corner information of the water jet.

[0100] Axis point positioning: Based on the extracted corner point information, a geometric analytical method is used to determine the precise location of key points on the axis of the divergent water jet.

[0101] 3D spatial coordinate calculation: Combining camera calibration parameters and image processing results, and using the principle of binocular stereo vision, the 3D spatial coordinates of a specific point on the axis of the diverging water jet are calculated by triangulation.

[0102] Collision detection and judgment mechanism: Based on the spatial position information of key points on the end effector of the robotic arm and the calculated coordinates of key points on the water jet axis, a mathematical model of the shortest distance between a straight line and a point is constructed. By comparing this shortest distance with a preset safety threshold, it is determined whether there is a potential risk of contact between the water jet and the clamping device.

[0103] This invention utilizes high-precision image acquisition and processing technology based on machine vision to achieve high-precision calculation of the spatial position of the axis point of a diverging water jet. The machine vision system can acquire and process images in real time, enabling real-time calculation of the spatial position of the axis point of the diverging water jet, and is suitable for dynamically changing diverging water jet scenarios. This invention is not only applicable to the measurement of diverging water jets but can also be extended to the spatial position calculation of other liquid or solid columns, demonstrating broad applicability. Attached Figure Description

[0104] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0105] Figure 1 This is a schematic diagram of the collision detection process of the present invention;

[0106] Figure 2 This is a schematic diagram of the DH parameters of the present invention;

[0107] Figure 3 A schematic diagram of the robotic arm linkage coordinate system established for this invention;

[0108] Figure 4 This is a schematic diagram of the collaborative dual robotic arm system model of the present invention;

[0109] Figure 5 This is a schematic diagram illustrating the transformation between coordinate systems in this invention;

[0110] Figure 6 This is a flowchart of the Zhang Zhengyou calibration method of the present invention;

[0111] Figure 7 This is a schematic diagram of the image before image processing according to the present invention;

[0112] Figure 8 This is a schematic diagram of the image after binarization according to the present invention;

[0113] Figure 9 This is a schematic diagram of the corner point extraction of divergent water jet images according to the present invention;

[0114] Figure 10 This is a schematic diagram of the image processing process of the present invention;

[0115] Figure 11 This is a schematic diagram illustrating the principle of binocular stereo vision positioning in this invention.

[0116] Figure 12 This is a mathematical model diagram of the distance between the straight line and point M in this invention. Detailed Implementation

[0117] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0118] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0119] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0120] like Figure 1 As shown, a safety protection method for water jet cutting collaborative dual robotic arms based on machine vision includes the following steps:

[0121] Step 1: Establish the kinematic model of the robotic arm, construct a fixed global world coordinate system and the local coordinate system of each link of the robotic arm, and perform kinematic modeling of the two robotic arms working in cooperation;

[0122] Step 2: Obtain the coordinates of key points of the robotic arm, transform the coordinates to associate the local coordinate system of each link of the robotic arm that was established in Step 1 with the global world coordinate system, and obtain the three-dimensional coordinate information of key point P1 on the end effector of the main robotic arm and point M from the center point of the end effector of the robotic arm in the global world coordinate system.

[0123] Step 3: Acquire the target image, calibrate the camera, and capture the target image;

[0124] Step 4: Process the image to obtain the outer contour image of the water jet;

[0125] Step 5: Obtain the coordinates of target point P2, extract target point P2 on the water jet axis, and calculate and determine the three-dimensional spatial position of target point P2 in the global world coordinate system.

[0126] Step 6: Calculate the shortest distance between the water jet axis and the robotic arm. Based on the points P1 and P2 obtained in Step 2 and Step 5, calculate the mathematical equation of the straight line L passing through P1 and P2 in the global world coordinate system, and calculate the shortest distance d between the straight line L and the water jet axis.

[0127] Step 7: Risk assessment. Establish a safe distance threshold r. By comparing the shortest distance d with the safe distance threshold r, if the shortest distance d is greater than the safe distance threshold r, return to step 3. If the shortest distance d is less than or equal to the safe distance threshold r, it is determined that there is a risk of contact and protective measures are taken.

[0128] The system of this invention consists of three parts: a computer control unit, a vision perception system, and a robotic arm execution system. The vision system acquires image information and feeds it back to the computer. The computer control unit uses image processing algorithms and data analysis to accurately identify and locate the spatial pose of the target object. The computer then sends commands to control the movement of the robotic arm based on its current pose.

[0129] like Figure 2 and 3 As shown, the DH method is used to establish the coordinate system of the two robotic arms.

[0130] In the diagram: a i = Along X i Axis, from Z i Move to Z i+1 The distance;

[0131] α i =around X i Axis, from Z i Rotate to Z i+1 Angle;

[0132] d i = Along Z i Axis, from X i-1 Move to X i The distance;

[0133] θ i = Around Z i Axis, from X i-1 Rotate to X i The angle.

[0134] according to Figure 2 The DH parameters and robot arm dimensions can be used to establish... Figure 3 Linkage coordinate system.

[0135] Since the two robotic arms are of the same type but different sizes, the kinematic modeling methods for the two robotic arms are the same, and only the kinematic modeling of the main arm is shown in the figure.

[0136] like Figure 4 As shown in the figure, the RGB-D camera is 1, the main arm of the robotic arm is 2, the water jet is 3, the clamping mechanism is 4, and the slave arm of the robotic arm is 5.

[0137] Based on the robotic arm link coordinate system established in step 1, the coordinate transformation is used to obtain... Figure 4 P1(X1,Y1,Z1) and M(X m ,Y m Z m The coordinates of the key point P1 in the world coordinate system are as follows: The key point P1 is located at the intersection of the water gun's axis and the axis of the last section of the robotic arm. The key point M is located at the center of the spherical enclosure of the clamping device.

[0138] The forward kinematics equations of the robotic arm can be obtained by matrix multiplication to obtain the transformation matrix between the links. i T j Let {j} represent the transformation matrix relative to coordinate system {i}. The overall transformation matrix of the robotic arm's terminal control mechanism relative to the base can be expressed as:

[0139] 0 T6 = 0 T1 1 T2 2 T3 3 T4 4 T5 5 T6 (1)

[0140] like Figure 5 As shown, coordinate transformation and camera calibration in binocular vision:

[0141] Camera calibration is the prerequisite and foundation for the successful application of machine vision in practice. Only by accurately calibrating the camera can the effective three-dimensional position information of the object be accurately calculated based on the acquired two-dimensional image sequence. First, based on the principle of pinhole imaging, the relationship between the image in the pixel coordinate system, the image coordinate system, and the camera coordinate system can be determined. The transformation relationship between these three coordinate systems can be described as shown in formula (2).

[0142]

[0143] Suppose p is a point in space, and its coordinates in the world coordinate system are... Its projection onto the pixel coordinate system has coordinates (u, v); This is the intrinsic parameter matrix of the camera. The extrinsic parameter matrix of the camera, along with the intrinsic parameter matrix, are collectively referred to as the camera matrix, both of which need to be obtained through camera calibration.

[0144] like Figure 6 As shown, the camera calibration method is Zhang Zhengyou's camera calibration method, and the specific method is as follows:

[0145] S1. Prepare a chessboard pattern and take pictures of it from different angles with a camera to obtain a set of images;

[0146] S2. Detect the corner points of the calibration board in the image, obtain the pixel coordinate values ​​of the corner points of the calibration board, and then calculate the physical coordinate values ​​of the corner points of the calibration board.

[0147] S3. Solve for the intrinsic and extrinsic parameter matrices;

[0148] S4. Optimize the above parameters using the LM (Levenberg-Marquardt) algorithm.

[0149] like Figure 7 and 8 As shown, image binarization processing:

[0150] From the images of the water jet cutting robot captured by the camera, given that the high-pressure water jet appears as a bright white area due to light scattering, its grayscale value is usually low. In order to accurately extract the outer contour information of the water jet, effectively remove cluttered background information, and preserve the image features of the high-speed water jet as completely as possible, we adopt an image binarization processing strategy.

[0151] Image binarization is a technique that converts an image into a single pixel value containing only two types of values, typically 0 and 255, representing black and white respectively. To separate the water jet contour pixels from the background pixels, a threshold T can be set to separate these two types of pixels. Any point (x,y) in the image where f(x,y)>T is the target point; otherwise, the point is the background point. This allows the target pixels to be distinguished from the background pixels. Figure 8 Image 6 in the middle is the binarized image of the water jet.

[0152] Edge detection involves performing edge detection on the binarized image to obtain the desired outer contour of the water jet and remove pixels that are not edge-defined.

[0153] The edge detection method is the Canny edge detection algorithm. Due to the influence of background and light, image binarization alone cannot completely separate the water jet contour from the background. Therefore, edge detection is required on the binarized image to obtain the required water jet outer contour and remove some non-edge pixels.

[0154] The specific method is as follows:

[0155] Step 1: First, perform Gaussian filtering on the input image to remove noise;

[0156] Step 2: On the smoothed grayscale image, use the Sobel operator to calculate the gradient magnitudes Gx(x,y) and Gy(x,y) in the horizontal and vertical directions;

[0157] Step 3: Non-maximum suppression. After calculating the gradient magnitude and direction, remove non-edge points.

[0158] Step 4: Use the Otsu algorithm to implement the adaptive selection of high and low thresholds in the Canny algorithm.

[0159] Steps for extracting straight lines using Hough transform.

[0160] After obtaining the edges of the image, which are not complete straight lines, we can perform straight line detection on the image based on the characteristic that the water jet contour is a straight line, in order to obtain the complete outer contour of the water jet.

[0161] Step 1: Obtain all points with a value of 1 in the binary image as target points, perform Hough transform on each target point to map it to the parameter space, and quantize the P and θ parameters into MP and Nθ equal parts.

[0162] Step 2: Based on the discretized MP and Nθ, establish a two-dimensional accumulator A with an initial value of 0.

[0163] Step 3: Calculate the coordinates of each point in the Cartesian coordinate system in polar coordinates. For each additional sinusoidal intersecting curve generated, increment the accumulator by 1.

[0164] Step 4: Select an appropriate threshold for judgment. When the value in the judgment accumulator is greater than the threshold, it proves that the matched line is the required straight line. Points on the same line appear as peak points in polar coordinates. Then calculate the length of the line and return it to Cartesian coordinates, thus completing the extraction of the straight line.

[0165] like Figure 9 As shown, Harris corner detection is used to extract the four corner points of the target:

[0166] After obtaining the outer contour of the water jet, it is necessary to extract the four corner points of the outer contour in order to calculate the position of a point P on the water jet axis in the image coordinate system.

[0167] Ideally, the high-pressure water jet from a water gun is cylindrical, but in reality, it tends to diverge, resembling a frustum. Figure 9As shown. In either state, when performing corner detection on an image, the center point of the four finally detected corner points, that is, the centroid of the quadrilateral, and the center point is on the axis of the water jet, all meet the purpose of the detection in this step. The center points of corner points (1, 2, 3, 4) and corner points (1, 2, 5, 6) are the same point and on the central axis of the water jet.

[0168] Harris corner detection is the gray-scale transformation generated by the local window moving on the image. Whether it is a corner point is judged by the degree of change of the window in each direction. Translate the image window (u, v) to generate the gray-scale change E(u, v).

[0169] After obtaining the image coordinates of the four corner points, the centroid coordinates of the water jet image in the image coordinate system can be obtained, and the coordinates P2 of P in the world coordinate system can be obtained using formula (2).

[0170] E(u,v) = ∑ x,y w(x,y)[I(x + u,y + v)-I(x,y)] 2 (3)

[0171] The above formula can be written as

[0172]

[0173] In the formula,

[0174]

[0175] The w function is the window function, and the M matrix is the partial derivative matrix. Among them, Ix and Iy are the partial derivative functions of the image in the horizontal and vertical directions respectively. According to the eigenvalue calculation method, 2 eigenvalues will be generated, and these 2 eigenvalues represent the intensity of the transformation of each pixel point. In the actual calculation process, the response value R of each corner point is calculated through the corner response function, and then whether it is a corner point is judged according to the R value.

[0176] If the set threshold Tr < R, then it is considered that this pixel point is a corner point in the image, otherwise it is a non-corner point in the image;

[0177] The mathematical definition of the corner response function is specifically as follows:

[0178] R = det(M)-k(traceM) 2 (6)

[0179] In the formula, k is an empirical constant, and the value range of k is between 0.04 and 0.06.

[0180] Such as Figure 10 and 11 As shown, obtain the three-dimensional coordinates of the target point P2:

[0181] Because the object of image processing is a diverging water jet, which resembles a frustum in space, the corner points obtained after image processing can be regarded as the four vertices A of a convex quadrilateral. i (u i ,v i ),like Figure 10 As shown, the centers of the four points lie on the axis. By calculating the centers of these four points, we can find a point on the axis.

[0182]

[0183] Based on the principle of binocular stereo vision positioning, such as Figure 11 As shown, the coordinate transformation yields the three-dimensional coordinates P2(X2,Y2,Z2) of the center point p in the world coordinate system. The desired P2 is a point on the axis of the divergent water jet.

[0184] like Figure 12 As shown, based on the obtained three-dimensional information of the target, the distance between the water jet and the center point of the actuator at the end of the main arm is calculated;

[0185] In the diagram: P1 is a key point on the water jet axis, P2 is a key point on the water jet axis, L is the water jet axis, d is the shortest path between point M and line L, and M is the center key point of the clamping device.

[0186] Depend on Figure 12 As shown, based on the obtained P1(X1,Y1,Z1) and P2(X2,Y2,Z2), a spatial straight line L can be obtained that coincides with the axis of the diverging water jet. The two-point equation of straight line L is:

[0187]

[0188] The M(X) obtained from step 1 m ,Y m Z m The shortest distance d between M and L can be calculated. The value of d can be calculated from the distance from a point to a line. Taking a point P1 on the line, we can obtain...

[0189]

[0190] Direction vector of line L

[0191]

[0192] but

[0193]

[0194] Where, a = (Y1 - Y) m(Z2-Z1)-(Y2-Y1)(Z1-Z m ),

[0195] b = (X1 - X) m (Z2-Z1)-(X2-X1)(Z1-Z m ),

[0196] c = (X1 - X m (Y2-Y1)-(X2-X1)(Y1-Y m ).

[0197] The shortest distance d between M and L can be obtained:

[0198]

[0199] Step 7: Conduct a contact risk assessment to determine whether the contact has occurred.

[0200] When d > r, the divergent water jet and the end-effector gripper are considered to be at a safe distance, and no collision occurs. When d ≤ r, the divergent water jet and the end-effector gripper are considered to be at a dangerous distance, and a collision may occur. In this case, measures can be taken to avoid a collision. Considering the accuracy of the vision system measurement and possible errors in other aspects, the value of r can be appropriately adjusted to prevent collisions.

[0201] In real-world operating environments, a water jet is ideally considered a regular cylinder. However, when a robot performs a cutting task, the water jet exhibits various divergent phenomena, with its axial vector position and length dynamically changing. Faced with this complex and ever-changing scenario, traditional positioning methods are inadequate in capturing and locating the position and direction of divergent water jets. A vision-based water-cutting collaborative dual-manipulator safety protection method integrates kinematic principles and machine vision algorithms to determine the spatial pose of divergent water jets. Particularly for those real-time changing and unpredictable divergent water jet patterns, this technology can quickly and accurately capture their dynamic characteristics. By locating the divergent water jet in real time, it can determine whether the device is in a dangerous state, thereby preventing high-speed jet cutting of the manipulator and achieving safe and reliable operation.

[0202] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0203] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A safety protection method for water jet cutting collaborative dual robotic arms based on machine vision, characterized in that... This includes the following steps: Step 1: Establish the kinematic model of the robotic arm, construct a fixed global world coordinate system and the local coordinate system of each link of the robotic arm, and perform kinematic modeling of the two robotic arms working in cooperation; Step 2: Obtain the coordinates of key points of the robotic arm, transform the coordinates to associate the local coordinate system of each link of the robotic arm that was established in Step 1 with the global world coordinate system, and obtain the three-dimensional coordinate information of key point P1 on the end effector of the main robotic arm and point M from the center point of the end effector of the robotic arm in the global world coordinate system. Step 3: Acquire the target image, calibrate the camera, and capture the target image; Step 4: Process the image to obtain the outer contour image of the water jet; Step 5: Obtain the coordinates of target point P2, extract target point P2 on the water jet axis, and calculate and determine the three-dimensional spatial position of target point P2 in the global world coordinate system. Step 6: Calculate the shortest distance between the water jet axis and the robotic arm. Based on the points P1 and P2 obtained in Step 2 and Step 5, calculate the mathematical equation of the straight line L passing through P1 and P2 in the global world coordinate system, and calculate the shortest distance d between the straight line L and the center point M of the robotic arm end effector. Step 7: Risk assessment. Establish a safe distance threshold r. By comparing the shortest distance d with the safe distance threshold r, if the shortest distance d is greater than the safe distance threshold r, return to step 3. If the shortest distance d is less than or equal to the safe distance threshold r, it is determined that there is a risk of contact and protective measures are taken.

2. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 1, characterized in that, The method for establishing the kinematic model of the robotic arm described in step 1 is as follows: 1.1 Establishing the coordinate system of the dual-arm linkage using the DH method a i = Along X i Axis, from Z i Move to Z i+1 distance, α i =around X i Axis, from Z i Rotate to Z i+1 Angle, d i = Along Z i Axis, from X i-1 Move to X i distance, θ i = Around Z i Axis, from X i-1 Rotate to X i Angle; 1.2 Establish the link coordinate system of the robotic arm based on the DH parameters and the dimensions of the robotic arm.

3. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 2, characterized in that, The method for obtaining the coordinates of the key points of the robotic arm in step 2 is as follows: 2.

1. Based on the robotic arm linkage coordinate system established in step 1, the key points P1(X1,Y1,Z1) and M(X) are determined through coordinate transformation. m ,Y m Z m The coordinate position in the world coordinate system. The key point P1 is located at the intersection of the water gun's axis and the axis of the last section of the robotic arm. The key point M is located at the center of the spherical enclosure of the clamping device; 2.

2. Obtain the forward kinematic equations of the robotic arm, and then obtain the transformation matrix between the links using matrix multiplication. i T j Let {j} represent the transformation matrix relative to coordinate system {i}. The overall transformation matrix of the robotic arm's terminal control mechanism relative to the base is expressed as: 0 T6= 0 T1 1 T2 2 T3 3 T4 4 T5 5 T6 (1)。 4. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 1, characterized in that, The method for calibrating the camera in step 3 is as follows: Based on the principle of pinhole imaging, the relationship between the image in the pixel coordinate system, the image coordinate system, and the camera coordinate system is determined. The transformation relationship between the three coordinate systems is described as shown in formula (2): Suppose p is a point in space, and its coordinates in the world coordinate system are... Its projection onto the pixel coordinate system has coordinates (u, v); This is the intrinsic parameter matrix of the camera. The extrinsic parameter matrix of the camera is obtained through camera calibration. The intrinsic parameter matrix and the extrinsic parameter matrix are collectively referred to as the camera matrix. The camera calibration method mentioned is Zhang Zhengyou's camera calibration method, and the specific method is as follows: S1. Prepare a chessboard pattern and take pictures of it from different angles with a camera to obtain a set of images; S2. Detect the corner points of the calibration board in the image, obtain the pixel coordinate values ​​of the corner points of the calibration board, and then calculate the physical coordinate values ​​of the corner points of the calibration board. S3. Solve for the intrinsic and extrinsic parameter matrices; S4. Optimize the above parameters using the LM (Levenberg-Marquardt) algorithm.

5. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 1, characterized in that, The image processing method in step 4 is as follows: 4.

1. Through image binarization processing, extract the outer contour information of the water jet from the water cutting robot arm operation image captured by the camera; set a threshold T to separate the two types of pixels. Any point (x,y) in the image with f(x,y)>T is the target point, otherwise the point is the background point. Distinguish between target pixels and background pixels, remove messy background information, and at the same time retain the image features of the high-speed water jet. 4.2 Edge detection: Perform edge detection on the binarized image to obtain the required outer contour of the water jet and remove pixels that are not edge-defined. The edge detection method is the Canny edge detection algorithm, and the specific method is as follows: Step 1: First, perform Gaussian filtering on the input image to remove noise; Step 2: On the smoothed grayscale image, use the Sobel operator to calculate the gradient magnitudes Gx(x,y) and Gy(x,y) in the horizontal and vertical directions; Step 3: Non-maximum suppression. After calculating the gradient magnitude and direction, remove non-edge points. Step 4: Use the Otsu algorithm to implement the adaptive selection of high and low thresholds in the Canny algorithm.

6. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 3, characterized in that, The method for obtaining the coordinates of the target point P2 in step 5 is as follows: 5.1 Extracting straight lines: Based on the characteristic that the water jet contour is a straight line, straight line detection is performed on the image to obtain the complete outer contour of the water jet. 5.2 Extract the four corner points of the target to obtain the outer contour of the water jet. Then extract the four corner points of the outer contour to calculate the position of a point P on the water jet axis in the image coordinate system. 5.3 Obtain the three-dimensional coordinates of the target point P2. The object of image processing is a divergent water jet, which is similar to a frustum in space. The corner points obtained after image processing are regarded as the four vertices of a convex quadrilateral. The center position of the four points is on the axis. Calculate the center position of these four points to obtain a point on the axis. Using the principle of binocular stereo vision positioning and coordinate transformation, the three-dimensional coordinates P2(X2,Y2,Z2) of the center point p in the world coordinate system are obtained. The desired P2 is a point on the axis of the divergent water jet.

7. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 6, characterized in that, The method for extracting straight lines is Hough transform extraction, and the specific method is as follows: Step 1: Obtain all points with a value of 1 in the binary image as target points. Perform a Hough transform on each target point to map it to the parameter space, and quantize the P and θ parameters into M. P and N θ Equal portions; The second step is to determine the discretized M... P and N θ Create a two-dimensional accumulator A with an initial value of 0; The third step is to calculate the coordinates of each point in the Cartesian coordinate system in polar coordinates. For each additional sinusoidal intersecting curve generated, increment the accumulator by 1. The fourth step is to select an appropriate threshold for judgment. When the value in the judgment accumulator is greater than the threshold, the matched line is the required straight line. Points on the same line appear as peak points in polar coordinates. Then, the length of the line is calculated and returned to Cartesian coordinates to complete the extraction of the straight line.

8. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 6, characterized in that, The method for extracting the four corner points of the target is Harris corner detection, and the specific method is as follows: The grayscale transformation caused by moving a local window across the image is used to determine whether a point is a corner by observing the degree of change in the window in various directions; translating the image window (u,v) produces a grayscale change E(u,v). The image coordinates of the four corner points are obtained, the centroid coordinates of the water jet image in the image coordinate system are obtained, and the coordinates P2 of P in the global world coordinate system are obtained using formula (2). E(u,v)=∑ x,y w(x,y)[I(x+u,y+v)-I(x,y)] 2 (3) The above formula is written as In the formula, The w function is a window function, and the M matrix is ​​a partial derivative matrix, where I x I y These are the partial derivatives of the image in the horizontal and vertical directions, respectively. Based on the eigenvalue calculation method, two eigenvalues ​​are generated. These two eigenvalues ​​represent the intensity of the transformation of each pixel. In the actual calculation process, the response value R of each corner point is calculated through the corner response function, and then the R value is used to determine whether it is a corner point. If the set threshold Tr < R, then it is considered that the pixel point is a corner point in the image; otherwise, it is a non-corner point in the image. The mathematical definition of the corner response function is specifically as follows: R=det(M)-k(traceM) 2 (6) In the formula, k is an empirical constant, and the value range of k is between 0.04 and 0.

06.

9. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 6, characterized in that, The method for obtaining the three-dimensional coordinates of the target point P2 is as follows: The corner points obtained after image processing are considered as the four vertices A of a convex quadrilateral. i (u i ,v i The centers of the four points lie on the axis. Calculating the center of these four points will find a point on the axis. Through the binocular stereo vision positioning principle and coordinate transformation, the three-dimensional coordinates P2(X2, Y2, Z2) of the center point p in the world coordinate system are obtained. The required P2 is a point on the axis of the divergent water jet.

10. The safety protection method for water cutting collaborative dual robotic arms based on machine vision according to claim 9, characterized in that, The method for calculating the shortest distance between the water jet axis and the robotic arm in step 6 is as follows: According to the obtained P1(X1, Y1, Z1) and P2(X2, Y2, Z2), a space straight line L parallel to the axis of the divergent water jet is obtained; The M(X) obtained from step 1 m ,Y m Z m ), calculate the shortest distance d between M and L; The value of d is calculated from the distance from the point to the line. Let's take a point P1 on the line and obtain... Direction vector of line L The two-point form equation of the straight line L: where a = (Y1 - Y m )(Z2 - Z1) - (Y2 - Y1)(Z1 - Z m ), b = (X1 - X m )(Z2 - Z1) - (X2 - X1)(Z1 - Z m ), c = (X1 - X m )(Y2 - Y1) - (X2 - X1)(Y1 - Y m ); Then The shortest distance d between M and L is obtained:

Images