Mechanical hand vision positioning system and method for ladle installation of long nozzle eyes

Abstract

The application belongs to the technical field of visual positioning, and particularly relates to a visual positioning system and method for eyes-in-hand of a long nozzle installed on a ladle, which comprises the following steps: collecting three-dimensional point cloud data of the long nozzle and a clamping end and performing pretreatment; extracting features of the long nozzle and the clamping end from the point cloud data, registering the features with pre-stored template features, and constructing a three-dimensional model; extracting center coordinates of a gripped surface of the long nozzle and a clamping surface in the three-dimensional model, calculating differences, and planning a path; calculating a registration transformation matrix of a long nozzle installation part to a ladle lower nozzle, aligning the point cloud data by using the transformation matrix, and planning a path; and sending the planned path to a control system. The application improves positioning accuracy and real-time self-adaptive capability of path planning by using an eyes-in-hand visual sensing mode, solves the problem of insufficient path planning, can effectively cope with position changes of a large ladle rotary table, reduces path deviation and docking failure risks, and improves automation degree and production efficiency.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning technology, and more specifically, relates to an eye-to-hand visual positioning system and method for installing long nozzles on steel ladles. Background Technology

[0002] In the steel metallurgical process, the ladle pouring and long nozzle replacement in the continuous casting workshop are critical operational steps. The smooth execution of this process directly impacts the quality of the molten steel and production efficiency. Traditional long nozzle replacement usually relies on manual operation, but this method suffers from low operational precision, low efficiency, and significant safety hazards. To address these issues and improve operational efficiency and safety, many steel plants have begun to adopt industrial robots for automated long nozzle replacement.

[0003] Existing industrial robots, through pre-programmed procedures and visual perception systems, automatically complete the tasks of grasping, positioning, and installing long sprues, replacing manual hydraulic grippers and achieving unmanned operation. These robots primarily rely on vision sensors and automatic control systems to accomplish their tasks. However, existing technologies still face some technical challenges and limitations in application, requiring further improvement and optimization.

[0004] Specifically, the current visual sensing technology used in industrial robotic arms for long nozzle replacement has the following problems:

[0005] 1. Limitations of Visual Sensors: Most existing vision systems employ binocular cameras and line laser technology. While binocular cameras are less expensive and can operate under various lighting conditions, their positioning accuracy is poor in distant or low-contrast scenes, and they cannot acquire accurate depth information on surfaces with little texture, affecting overall positioning accuracy. Furthermore, the positioning algorithms of binocular cameras are complex and computationally expensive. While line laser technology offers high precision and resolution, it can only acquire depth data for a single line, making it difficult to directly obtain complete 3D information. It requires movement of the sensor or target to acquire global data, which not only increases operational complexity but also raises equipment costs.

[0006] 2. "Outside-of-the-eye" vision sensor mounting method: Most vision sensors are mounted on the outside of the robotic arm. While this "outside-of-the-eye" method offers good stability, the fixed viewing angle prevents it from adjusting with the arm's movement, thus limiting operational flexibility. When handling tasks requiring different angles or positions, the fixed viewing angle may not provide sufficient information.

[0007] 3. Insufficient path planning and real-time adaptive capabilities: Existing robotic arms typically employ fixed path planning methods, lacking the ability to adapt to dynamic changes. For example, when the rotation and lifting positions of the large-bag turntable change, the robotic arm cannot adjust its path in real time to adapt to the new position. In such cases, path deviations or docking failures are prone to occur, often requiring recalibration before operation or reliance on manual intervention, which significantly reduces the system's automation level and operational efficiency.

[0008] Therefore, steel plants urgently need an improved method to solve the above problems, such as using the "eye-in-hand" visual positioning method to improve the operational flexibility and accuracy of robotic arms during the replacement of long nozzles in steel ladles. Summary of the Invention

[0009] The purpose of this invention is to provide an eye-to-hand manipulator visual positioning system and method for installing long nozzles on steel ladles, so as to solve the problems of limited vision sensors, fixed viewing angle limitations, and insufficient path planning and real-time adaptive capabilities in the above-mentioned background art.

[0010] To achieve the above objectives, in one aspect, the present invention provides the following technical solution: a visual positioning method for an eye-to-hand manipulator for installing long nozzles on steel ladles, used to control a manipulator equipped with a camera device on its mechanical gripping end, the manipulator being used to grip the long nozzle that connects to the ladle's outlet, the method comprising:

[0011] Step 1: Collect three-dimensional point cloud data of the long water inlet and clamping end, and preprocess the three-dimensional point cloud data;

[0012] Step 2: Extract the features of the long sprue and clamping end from the preprocessed point cloud data, register the extracted features with their respective pre-stored template features, transform the registered point cloud data as a whole, and construct a 3D model of the long sprue and clamping end.

[0013] Step 3: Extract the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end in the 3D model, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference.

[0014] Step 4: Based on the coordinate positions of the ladle outlet and the long nozzle installation part, calculate the registration transformation matrix from the long nozzle installation part to the ladle outlet. Use the transformation matrix to align the point cloud data of the long nozzle installation part and the ladle outlet, and use the path planning algorithm to calculate the path from the centroid position of the long nozzle installation part to the target position of the outlet at the clamping end.

[0015] Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long water inlet, and the planned path from the long water inlet installation part to the ladle outlet to the mechanism control system.

[0016] Furthermore, in step 2, the construction of the three-dimensional model is based on the registered feature points of the long sprue and the clamping end. The transformation matrix between the long sprue and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the long sprue are calculated respectively. The transformation matrix is ​​used to align the point cloud of the long sprue and the point cloud of the clamping end to the template coordinate system respectively, and merge them into a complete three-dimensional model.

[0017] Furthermore, in step 3, the center coordinates of the gripping surface of the long nozzle and the gripping surface of the clamping end are obtained by the centroid calculation method.

[0018] Furthermore, the centroid calculation method in step 3 includes:

[0019] Summing the coordinates of each feature point in the feature point set of the grasped surface, and then dividing the sum of the coordinates of all feature points by the total number of feature points, yields the centroid coordinates of the grasped surface of the long sprue:

[0020]

[0021] Summing the coordinates of each feature point in the feature point set of the gripping end's grasping surface, and then dividing the sum of the coordinates of all feature points by the total number of feature points, yields the centroid coordinates of the gripping end's grasping surface:

[0022]

[0023] In the formula: Represents the centroid coordinates of the surface being grabbed by the long water inlet; The coordinates of the centroid of the gripping surface are represented by n; n represents the total number of matched feature point pairs; p i q represents the i-th feature point in the point cloud of Changshuikou; i This represents the i-th feature point that matches the pre-stored template of the long water outlet.

[0024] Furthermore, in step 4, the alignment feature of the long nozzle installation part is the normal vector of the installation surface, and the alignment feature of the ladle outlet is the normal vector of the outlet. By calculating the rotation matrix and translation vector between the normal vectors, a homogeneous transformation matrix from the long nozzle installation part to the ladle outlet is formed.

[0025] On the other hand, the present invention also provides an eye-in-the-hand manipulator visual positioning system for installing long nozzles on steel ladles, the eye-in-the-hand manipulator visual positioning system being used to implement an eye-in-the-hand manipulator visual positioning method for installing long nozzles on steel ladles, comprising:

[0026] Visual acquisition equipment is used to acquire three-dimensional point cloud data of the long sprue and clamping end;

[0027] The clamping unit is used to clamp the long nozzle and perform the gripping, moving and docking operations of the long nozzle with the ladle outlet.

[0028] The processor is used to preprocess the 3D point cloud data of the long sprue and the clamping end, extract the features of the long sprue and the clamping end from the preprocessed point cloud data, register them with their respective pre-stored template features, determine the transformation relationship between the extracted features and the template features, extract the center coordinates of the gripped surface of the long sprue and the gripping surface of the clamping end in the 3D model according to the coordinate positions of the ladle sprue and the long sprue installation part, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and calculate the registration transformation matrix from the long sprue installation part to the ladle sprue according to the coordinate positions of the ladle sprue and the long sprue installation part, and use the transformation matrix to align the point cloud data of the long sprue installation part and the ladle sprue.

[0029] The model building unit is used to transform the registered point cloud data as a whole and build a 3D model of the long nozzle and clamping end.

[0030] Furthermore, the processor is also used to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference using a path planning algorithm, and to calculate the path from the centroid position of the clamping end of the long nozzle installation part to the target position of the outlet using a path planning algorithm, and to send the planned path from the gripping surface of the clamping end to the gripped surface of the long nozzle, and the planned path from the long nozzle installation part to the outlet of the ladle to the mechanism control system.

[0031] Furthermore, the visual acquisition device is mounted on the clamping unit using an eye-to-hand method.

[0032] Furthermore, the visual acquisition equipment includes a 3D structured light camera.

[0033] Furthermore, the clamping unit includes a robotic arm and a clamping end integrated with the robotic arm.

[0034] The beneficial effects of this invention are:

[0035] First, the eye-to-hand robotic arm vision positioning method for installing long nozzles on steel ladles, as described in this invention, achieves high-precision automatic replacement of long nozzles through intelligent path planning and adaptive control. Utilizing 3D point cloud data acquisition and feature extraction, the system calculates and adjusts the robotic arm path in real time to ensure accurate grasping and installation, adapting to dynamic environmental changes. This method significantly improves operational accuracy, success rate, and flexibility, reduces manual intervention, enhances automation levels, and thereby optimizes steel quality and production efficiency, reduces operational risks and costs, and provides an efficient and safe solution for steel metallurgical production.

[0036] Secondly, in the preferred implementation, the present invention calculates and applies the transformation matrices between the long nozzle and the clamping end and the pre-stored template, as well as the transformation matrix between the clamping end and the long nozzle, so as to accurately align the point cloud data to the template coordinate system and construct a complete three-dimensional model. This accurate alignment method enhances the consistency and stability of the long nozzle and the clamping end during the gripping and installation process, and reduces deviations and error accumulation during operation.

[0037] Third, in the preferred implementation, the present invention uses the centroid calculation method to obtain the center coordinates of the long nozzle and the gripping surface of the clamping end, and realizes the alignment operation between the long nozzle installation part and the ladle nozzle through normal vector calculation. Combined with the calculation of rotation matrix and translation vector, the path and posture adjustment of the clamping end are accurately planned. This method ensures the accuracy of gripping and docking during the installation of the long nozzle, reduces the deviation in path planning, and improves the adaptability and operating efficiency of the robot under complex working conditions.

[0038] Fourth, the eye-to-hand robotic vision positioning system for installing long nozzles on steel ladles of this invention integrates a vision acquisition device, a gripping unit, a processor, and a model building unit. A unified path planning algorithm coordinates the work of each module, ensuring efficient operation throughout the entire process from data acquisition, feature extraction, model building to path execution. The robotic arm in the gripping unit works collaboratively with the gripping end, combined with the processor's execution of path planning instructions, to achieve precise docking from the long nozzle gripping surface to the target position of the steel ladle outlet. This highly integrated design reduces the complexity of intermediate steps, improves the system's automation level and operational efficiency, and lowers operating costs and the need for manual intervention.

[0039] Fifth, in the preferred implementation, the visual acquisition device of the present invention uses a 3D structured light camera and is mounted on the clamping unit in an eye-on-hand manner, realizing dynamic real-time three-dimensional perception of the long sprue and the clamping end. This method greatly improves the accuracy of the long sprue grasping, moving and docking process, while allowing the system to quickly adjust the operation path according to real-time environmental changes, enhancing the flexibility and adaptability of the operation. Attached Figure Description

[0040] Figure 1 This is a flowchart of the eye-in-hand manipulator visual positioning method for installing long nozzles in steel ladles according to the present invention;

[0041] Figure 2 This is a three-dimensional structural diagram of the unmanned robotic arm with eyes in the hand for steel ladle casting according to the present invention.

[0042] Among them, 1-vision acquisition device; 2-clamping unit; 20-clamping end; 21-robotic arm. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this application, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0044] The directional terms such as above, below, left, right, front, and back used in this application are based on the positional relationships shown in the attached drawings. Different attached drawings may result in different positional relationships, therefore they should not be interpreted as limitations on the scope of protection.

[0045] In this application, the terms "installation," "connection," "interlocking," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, or a connection that allows communication between components. They can also refer to a direct connection or an indirect connection through an intermediate medium. They can refer to the internal connection of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0046] As per the instruction manual Figure 1 This invention describes a visual positioning method for an eye-in-the-hand manipulator used for installing long nozzles on steel ladles, comprising:

[0047] Step 1: Collect three-dimensional point cloud data of the long water inlet and clamping end, and preprocess the three-dimensional point cloud data.

[0048] It should be noted that the long sprue nozzles are placed on a designated long sprue nozzle shelf. During system initialization or planning, the coordinates of the long sprue nozzle placement area are preset and stored in the robot's control system. These coordinates can be derived from the process layout diagram or measured manually, ensuring their accuracy meets operational requirements. The system reads the preset coordinates of the long sprue nozzle placement area from storage and sets these coordinates as the target position for the robot's movement. The robot uses its own path planning module (such as A*, Dijkstra's algorithm, or other path planning algorithms) to calculate the optimal path from its current position to the preset coordinates and moves along the planned path to the long sprue nozzle placement area.

[0049] Specifically, a visual acquisition device is used to photograph the long water inlet within the placement area, acquiring its 3D point cloud data. The top of the long water inlet has a gripping surface and a mounting part. The gripping surface is used to connect with the gripping surface of the clamping end, and the mounting part is used to connect to the drain outlet. The 3D point cloud data includes the point cloud data of the gripping surface and the mounting part. The visual acquisition device is mounted on the robotic arm in an "eye-on-hand" manner, thus acquiring the 3D point cloud data of the clamping end simultaneously with the acquisition of the long water inlet's 3D point cloud data.

[0050] In the embodiments of this application, wavelet denoising is used to preprocess the acquired 3D point cloud data of the long sprue and the clamping end. Taking the preprocessing of the 3D point cloud data of the long sprue as an example:

[0051] First, wavelet transform is performed on the 3D point cloud data of Changshuikou to decompose the data into signal components of different frequencies. Wavelet decomposition can analyze point cloud data at different spatial frequencies, helping to identify the global morphology and local details of the data. The decomposed signals include low-frequency components and high-frequency components. The low-frequency components reflect the main morphology and global structure of Changshuikou, while the high-frequency components contain details, local changes, and noise information. Through wavelet transform, noise and useful features in the point cloud data can be separated, providing a basis for subsequent noise removal.

[0052] Secondly, the high-frequency components (containing details and noise) obtained from the decomposition are thresholded. An appropriate threshold is set (through experience, statistical methods, or adaptive algorithms) to filter the signal amplitude in the high-frequency components. Noise components below the threshold are zeroed or reduced, while components above the threshold (important details) are retained, thereby removing noise and reducing random errors and irregular fluctuations in the point cloud data.

[0053] Finally, the low-frequency and high-frequency components that have undergone thresholding are recombined, and the frequency components are synthesized into complete point cloud data through inverse wavelet transform. During the recombination process, the original spatial features of the Changshuikou point cloud data are accurately restored, ensuring that the denoised data retains the original geometric structure and positional relationships, thus generating denoised point cloud data.

[0054] Step 2: Extract the features of the long nozzle and clamping end from the preprocessed point cloud data, register the extracted features with their respective pre-stored template features, transform the registered point cloud data as a whole, and construct the three-dimensional model of the long nozzle and clamping end.

[0055] Step 2 specifically includes:

[0056] Step 2.1: Extract features from the point cloud data of the denoised long nozzle and clamping end respectively.

[0057] The features extracted from the long nozzle include edge features, hole features, gripping surface features, and mounting features. Edge features extract the edges and corner areas of the long nozzle surface, defining its overall outline. Hole features identify the locations of mounting holes on the long nozzle; these holes are important reference points during registration. The gripping surface refers to the surface on the long nozzle specifically designed to mate with the clamping end. These features typically include planes, recessed areas, or other structures that facilitate clamping. Mounting features extract the mounting plane, mounting holes, and mounting edges; these features are the core components for mating with the clamping end.

[0058] The features extracted from the gripping end include gripping surface features and gripping position features. Gripping surface features identify the edges and contact points of the gripping surface, serving as an alignment reference during the gripping process. Gripping position features extract the geometric features of the gripping position, including the specific location of the gripping surface and the features of the fixture surface.

[0059] In the embodiments of this application, the SIFT (Scale-Invariant Feature Transform) algorithm is used to identify the geometric feature points of the long nozzle and the clamping end, respectively. Taking the feature extraction of the long nozzle as an example:

[0060] First, by using a Gaussian filter to blur the point cloud data at different levels, images at multiple scales are constructed, allowing for the analysis of feature points at different scales. This is equivalent to observing the image through "blurred lenses" of varying intensities, thereby identifying feature points that remain distinct at various scales.

[0061] Then, within these different levels of the image, we search for "bright spots" or "dark spots" that stand out more than their surroundings; these are the geometric feature points of interest. These feature points remain unchanged across different scales and orientations. We also create an "identity card" for each feature point, capturing the image's surrounding trends (similar to the direction of changes in brightness), thus better describing the nature of the feature point.

[0062] Finally, the extracted feature points are filtered, removing those with low contrast, insufficient visibility, or those located at edges or prone to errors. This helps reduce noise and unstable feature points. The filtered feature point set retains the main geometric information of the long nozzle, such as the edges, the location of surface changes, the unique features of the grasped surface, and the installation location. These high-quality feature points provide a reliable data foundation for subsequent matching and registration.

[0063] Step 2.2: Use nearest neighbor search to initially align the feature points of the long water inlet and the clamping end with their respective pre-stored template feature points. At the same time, initially align the feature points of the clamping end and the long water inlet. Then use the ICP (Iterative Nearest Point) algorithm to completely overlap the features of the initially aligned long water inlet and the clamping end with the template features. At the same time, completely overlap the feature points of the clamping end and the long water inlet.

[0064] Specifically, nearest neighbor search finds the closest point in the template for each feature point by comparing the Euclidean distances between feature points, thus achieving approximate alignment. For each feature point, the minimum distance to the template feature point set is calculated. The point pairs with the minimum distance are considered matching pairs, and these pairings are marked as the basis for preliminary alignment. Based on the preliminary matching point pairs, a preliminary rigid body transformation matrix, including translation and rotation, is calculated and applied to the point cloud data of the long nozzle and clamping end to initially align it with the template point cloud.

[0065] The ICP (Iterative Closest Point) algorithm performs fine-grained registration of feature points between the long nozzle and the template based on initial alignment. Through multiple iterations, the ICP algorithm continuously minimizes the error between matching point pairs, adjusts the point cloud pose and position of the long nozzle and the clamping end, and achieves precise docking of the long nozzle point cloud data with the long nozzle pre-stored template, precise docking of the clamping end point cloud data with the clamping end pre-stored template, and precise docking of the matching point clouds of the clamping end and the long nozzle.

[0066] Step 2.3: Based on the registered feature points of the long sprue and the clamping end, calculate the transformation matrix between the long sprue and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the long sprue. Using the transformation matrices, align the point cloud of the long sprue and the point cloud of the clamping end to the template coordinate system as a whole, and align the gripping position of the clamping end to the gripped surface of the long sprue. Merge the point clouds of the long sprue and the clamping end, and construct the three-dimensional model of the long sprue and the clamping end using the mesh generation method.

[0067] In the embodiments of this application, the transformation matrix of the long water inlet and its pre-stored template is used as an example to illustrate the calculation process of the transformation matrix:

[0068] S1: Calculate the centroids of the long-sprue point cloud and the template point cloud for the matched feature point pairs respectively.

[0069] The centroid of the long nozzle is expressed by the following formula:

[0070]

[0071] The centroid of the pre-stored template at the long nozzle is expressed by the following formula:

[0072]

[0073] In the formula: n represents the total number of matched feature point pairs; p i q represents the i-th feature point in the point cloud of Changshuikou; i This indicates that the pre-stored template of the long water outlet contains p. i The i-th matching feature point.

[0074] S2: Subtract the corresponding centroid from each feature point in the long water inlet and the long water inlet pre-stored template to obtain the centroid-free point cloud coordinates.

[0075] The centroid-free point cloud representation of the long water inlet is:

[0076] The centroid-free long nozzle pre-stored template point cloud representation is as follows:

[0077] S3: Compute the covariance matrix H of the point cloud obtained by centroid removal from the point cloud of the long water inlet and the template point cloud, perform singular value decomposition (SVD) on the covariance matrix H, and use the result of singular value decomposition to calculate the rotation matrix R.

[0078] S5: Using the rotation matrix R and the centroid position of the point cloud, calculate the translation vector T to translate the centroid of the measured point cloud at Changshuikou to the centroid position of the pre-stored template point cloud at Changshuikou.

[0079] S6: Combine the rotation matrix R and the translation vector T to construct a 4x4 homogeneous transformation matrix, and obtain the transformation matrix of the long water inlet and its pre-stored template.

[0080] Step 3: Extract the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end in the 3D model, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference.

[0081] Let the feature point set of the surface to be crawled be {p1, p2, ..., p...} n}, where each feature point p i Having coordinates (x) i ,y i ,z i Using the centroid calculation method, the centroids of the feature points of the grasped surface are calculated. The coordinates of all feature points are summed, and then divided by the total number of feature points n. The centroid calculation method for the grasped surface of the long gate is determined by the following formula:

[0082]

[0083] According to formula (3), the center coordinates p of the grasped surface are obtained. install =(x instart ,y instart ,z instart ).

[0084] Similarly, let the feature point set of the grasping surface be {q1,q2,……,q}. n}, where each feature point q i Having coordinates (x) j ,y j ,z jUsing the centroid calculation method, the coordinates of all feature points on the gripping surface are summed, and then divided by the total number of feature points m. The centroid of the gripping surface at the clamping end is determined by the following formula:

[0085]

[0086] According to formula (4), the center coordinate q of the gripping surface at the clamping end is obtained. start =(x start ,y start ,z start ).

[0087] The center coordinate p of the grabbed surface install and the center coordinates q of the gripping end. start Used for path planning and docking operations at the clamping end.

[0088] Path planning algorithms include A*, Dijkstra, RRT*, etc. In the embodiments of this application, taking the A* algorithm as an example, the path calculation process from the current position of the gripping end to the position of the gripped surface of the long nozzle includes:

[0089] S1: Model the workspace of the gripper operation as a graph, where nodes represent passable locations and edges represent possible movement paths. Let the current position of the gripper be the starting node q, and the coordinates of point q be (x, y). start ,y start ,z start Let the center of the surface being grabbed by the long water outlet be the target node, and let the coordinates of point p be (x, y). instart ,y instart ,z instart ), calculate the difference in center coordinates between the gripping position of the clamping end and the gripped surface of the long nozzle.

[0090] The coordinate difference between the gripping position of the clamping end and the gripped surface of the long nozzle is:

[0091] Δp=p install -q start =[x instart -x start ,y instart -y start ,z instart -z start ]

[0092] S2: Create an initialization structure, which includes an open list, a closed list, a g list, and an f list. Select the node with the smallest f value as the current node, update the cost of adjacent nodes, and continue until the target node p is found.

[0093] It should be noted that the total estimated cost for each node is f = g + h, where h is the cost of the heuristic estimation, provided by the coordinate difference Δp. The initial f value of the starting point q is equal to its heuristic estimated cost, and the initial f value of other nodes is infinity.

[0094] S3: Starting from the target node p, trace the path backward through the predecessor node table until returning to the starting point q. Arrange the nodes in the tracing order to form the shortest path from the starting point to the target point.

[0095] Through these steps, the A* algorithm helps the gripping end efficiently find and execute the optimal path from the current position to the gripped surface of the long nozzle, ensuring the accuracy of gripping and installation operations.

[0096] Step 4: Based on the coordinate positions of the ladle outlet and the long nozzle installation part, calculate the registration transformation matrix from the long nozzle installation part to the ladle outlet. Use the transformation matrix to align the point cloud data of the long nozzle installation part and the ladle outlet, and use the path planning algorithm to calculate the path from the centroid position of the long nozzle installation part to the target position of the outlet at the clamping end.

[0097] It should be noted that the actual coordinates of the ladle outlet were calculated before performing step 1 of this application. The calculation method and the coordinate calculation method of the long outlet installation part can be referred to the calculation principle of steps 1-2 of this application, and will not be repeated here.

[0098] Specifically, step 4 includes:

[0099] Step 4.1: With the centroid of the gripping surface aligned with the centroid of the gripped surface of the long nozzle, obtain the coordinates of the centroid of the long nozzle installation part and the measured coordinates of the ladle outlet, and calculate the translation vector from the long nozzle installation part to the ladle outlet.

[0100] The coordinates of the centroid of the long nozzle installation section are c. install =(x instart ,y instart ,z instart The measured coordinates p of the ladle drain outlet tundish =(x tundish ,y tundish ,z tundish The translation vector from the long nozzle installation section to the ladle outlet is:

[0101] T align =p tundish -c install [x tundish -x instart ,y tundish -y instart ,z tundish -z instart

[0102] Step 4.2: Extract the alignment features of the long nozzle installation section and the ladle outlet, and calculate the rotation matrix between them.

[0103] Specifically, the normal vector n of the mounting surface is extracted from the 3D point cloud of the long water inlet mounting section. install Extract the normal vector n of the drain outlet from the 3D point cloud or pre-stored template of the drain outlet. tundish The rotation matrix is ​​calculated using the Rodriguez formula:

[0104]

[0105] In the formula: θ represents the normal vector n install and n tundish The included angle between them (i.e., the amount by which the mounting part of the long nozzle needs to be rotated to align with the ladle outlet), where u represents the rotation axis.

[0106] The rotation axis u is the normal vector n install and normal vector n tundish The product of n install and n tundish If they are parallel, then u is the zero vector, and no rotation is needed.

[0107] Step 4.3: Combine the translation vector and the rotation matrix to form the final homogeneous transformation matrix.

[0108] The homogeneous transformation matrix is:

[0109]

[0110] Step 4.4: Set the current position of the clamping end as the starting point of the path planning, and the position of the ladle drain outlet as the target position of the path planning. The path planning algorithm calculates the path from the starting point to the target point.

[0111] In the embodiments of this application, step 4.4 can use the same A* programming algorithm as step 3, and the specific process will not be described in detail.

[0112] Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long water inlet, and the planned path from the long water inlet installation part to the ladle outlet to the mechanism control system.

[0113] Specifically, the path point sets for the capture and docking paths are formatted into standard data packets (e.g., JSON, XML, or custom binary formats) to ensure good compatibility and transmission stability. Clear identifiers (such as "capture path data" and "docking path data") are added to the data packets to ensure the control system can correctly identify the path type and purpose. The mechanism control system receives the path data packets, parses their contents, and extracts the path point sets and related instructions. Following the order of the path point sets and the instructions, the control system directs the gripping end to perform the capture and docking operations.

[0114] Refer to the instruction manual appendix Figure 2 The present invention also describes an eye-in-the-hand manipulator visual positioning system for installing long nozzles on steel ladles. The eye-in-the-hand manipulator visual positioning system is used to realize the eye-in-the-hand manipulator visual positioning method for installing long nozzles on steel ladles, including a visual acquisition device 1, a clamping unit 2, a processor, and a model building unit.

[0115] The visual acquisition device 1 is installed on the clamping unit 2 in an eye-to-hand manner to acquire three-dimensional point cloud data of the long sprue and the clamping end.

[0116] In the embodiments of this application, the visual acquisition device is mounted above the clamping unit 2 in an "eye-on-hand" manner. The visual acquisition device includes binocular cameras, line laser devices, 3D structured light cameras, etc. The present invention preferably uses a 3D structured light camera. The 3D structured light camera can quickly and accurately acquire the three-dimensional data of an object by projecting a specific pattern and capturing deformed images, combined with high-precision calculation and real-time processing technology.

[0117] The clamping unit 2 includes a robotic arm 21 and a clamping end 20 integrated with the robotic arm 21. The clamping end 20 has a clamping surface that matches the clamping surface of the long nozzle. The clamping unit 2 is used to clamp the long nozzle and perform the gripping, moving and docking operations of the long nozzle with the ladle outlet.

[0118] The processor preprocesses the 3D point cloud data of the long sprue and clamping end. It extracts features of the long sprue and clamping end from the preprocessed point cloud data and registers them with their respective pre-stored template features, determining the transformation relationship between the extracted features and the template features. Based on the coordinate positions of the ladle outlet and the long sprue installation part, it extracts the center coordinates of the gripping surface of the long sprue and the gripping surface of the clamping end in the 3D model. It calculates the difference between the center coordinates of the gripping surface and the clamping surface, and uses a path planning algorithm to calculate the clamping end based on the coordinate difference. The path from the current position to the gripped surface of the long nozzle, and the registration transformation matrix from the long nozzle installation part to the ladle nozzle, are calculated based on the coordinate positions of the ladle nozzle outlet and the long nozzle installation part. The point cloud data of the long nozzle installation part and the ladle nozzle outlet are aligned using the transformation matrix. The path planning algorithm is then used to calculate the path from the centroid position of the long nozzle installation part to the target position of the nozzle outlet at the clamping end. The planned paths from the gripping surface of the clamping end to the gripped surface of the long nozzle, and from the long nozzle installation part to the ladle nozzle outlet, are sent to the mechanism control system.

[0119] The model building unit is used to transform the registered point cloud data as a whole and build a 3D model of the long nozzle and clamping end.

[0120] This invention achieves high-precision automatic replacement of long nozzles during ladle installation through intelligent path planning and adaptive control. It employs 3D point cloud data and centroid calculation for precise alignment and path planning, improving the accuracy and stability of gripping and installation. Integrating a vision acquisition device, clamping unit, and processor, the device uses real-time perception and dynamic adjustment of the operation path to enhance automation, operational efficiency, and system flexibility, thereby optimizing molten steel quality and reducing costs.

[0121] The above descriptions are merely embodiments of the present invention, and common knowledge regarding specific structures and characteristics of the solutions is not described in detail here. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present invention is defined by the appended claims rather than the foregoing description. Therefore, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A vision-based positioning method for an eye-to-hand manipulator used for installing long nozzles on steel ladles, comprising controlling a manipulator equipped with a camera at its mechanical gripping end, the manipulator being used to grip the long nozzle that is connected to the ladle's outlet, characterized in that... The methods include: Step 1: Collect three-dimensional point cloud data of the long water inlet and clamping end, and preprocess the three-dimensional point cloud data; Step 2: Extract the features of the long sprue and clamping end from the preprocessed point cloud data, register the extracted features with their respective pre-stored template features, transform the registered point cloud data as a whole, and construct a 3D model of the long sprue and clamping end. Step 2.1: Extract features from the point cloud data of the denoised long nozzle and clamping end respectively; The features extracted from the long nozzle include edge features, hole features, gripped surface features, and mounting features; the features extracted from the clamping end include gripping surface features and clamping position features. Step 2.2: Use nearest neighbor search to initially align the feature points of the long water inlet and the clamping end with their respective pre-stored template feature points. At the same time, initially align the feature points of the clamping end and the long water inlet. Then use the ICP algorithm to make the features of the initially aligned long water inlet and the clamping end completely overlap with the template features. At the same time, make the feature points of the clamping end and the long water inlet completely overlap. Step 2.3: Based on the registered feature points of the long sprue and the clamping end, calculate the transformation matrix between the long sprue and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the long sprue. Using the transformation matrix, align the point cloud of the long sprue and the point cloud of the clamping end to the template coordinate system as a whole, and align the gripping position of the clamping end to the gripped surface of the long sprue. Merge the point clouds of the long sprue and the clamping end, and construct the three-dimensional model of the long sprue and the clamping end using the mesh generation method. The transformation matrix of the long water inlet and its pre-stored template includes: calculating the centroids of the long water inlet point cloud and the template point cloud for each matched feature point pair; subtracting the corresponding centroid from each feature point in the long water inlet and the pre-stored template to obtain the centroid-free point cloud coordinates; calculating the covariance matrix H of the centroid-free point cloud of the long water inlet and the template point cloud, performing singular value decomposition on the covariance matrix H, and using the result of the singular value decomposition to calculate the rotation matrix R; using the rotation matrix R and the centroid position of the point cloud, calculating the translation vector T, and translating the centroid of the measured point cloud of the long water inlet to the centroid position of the pre-stored template point cloud of the long water inlet; combining the rotation matrix R and the translation vector T to construct a 4x4 homogeneous transformation matrix to obtain the transformation matrix of the long water inlet and its pre-stored template. Step 3: Extract the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end in the 3D model, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference. Step 4: Based on the coordinate positions of the ladle outlet and the long nozzle installation part, calculate the registration transformation matrix from the long nozzle installation part to the ladle outlet. Use the transformation matrix to align the point cloud data of the long nozzle installation part and the ladle outlet, and use the path planning algorithm to calculate the path from the centroid position of the long nozzle installation part to the target position of the outlet at the clamping end. Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long water inlet, and the planned path from the long water inlet installation part to the ladle outlet to the mechanism control system.

2. The eye-to-eye manipulator visual positioning method for installing long nozzles on steel ladles according to claim 1, characterized in that, In step 3, the center coordinates of the gripping surface of the long nozzle and the gripping surface of the clamping end are obtained by the centroid calculation method.

3. The eye-to-eye manipulator visual positioning method for installing long nozzles on steel ladles according to claim 2, characterized in that, The centroid calculation method in step 3 includes: Summing the coordinates of each feature point in the feature point set of the grasped surface, and then dividing the sum of the coordinates of all feature points by the total number of feature points, yields the centroid coordinates of the grasped surface of the long sprue: ； Summing the coordinates of each feature point in the feature point set of the gripping end's grasping surface, and then dividing the sum of the coordinates of all feature points by the total number of feature points, yields the centroid coordinates of the gripping end's grasping surface: ； In the formula: Indicates the centroid coordinates of the surface being grabbed by the long water inlet; The coordinates of the centroid of the gripping surface are represented; n represents the total number of matched feature point pairs. Let i represent the i-th feature point in the point cloud of Changshuikou; This represents the i-th feature point that matches the pre-stored template in the long water inlet.

4. The eye-to-eye manipulator visual positioning method for installing long nozzles on steel ladles according to claim 1, characterized in that, In step 4, the alignment feature of the long nozzle installation part is the normal vector of the installation surface, and the alignment feature of the ladle outlet is the normal vector of the outlet. By calculating the rotation matrix and translation vector between the normal vectors, a homogeneous transformation matrix from the long nozzle installation part to the ladle outlet is formed.

5. A vision positioning system for an eye-on manipulator with a long sprue nozzle mounted on a steel ladle, characterized in that, The eye-in-the-hand manipulator visual positioning system is used to implement the eye-in-the-hand manipulator visual positioning method for installing long nozzles on steel ladles as described in claim 1, including: Visual acquisition equipment is used to acquire three-dimensional point cloud data of the long sprue and clamping end; The clamping unit is used to clamp the long nozzle and perform the gripping, moving and docking operations of the long nozzle with the ladle outlet. The processor is used to preprocess the 3D point cloud data of the long sprue and the clamping end, extract the features of the long sprue and the clamping end from the preprocessed point cloud data, register them with their respective pre-stored template features, determine the transformation relationship between the extracted features and the template features, extract the center coordinates of the gripped surface of the long sprue and the gripping surface of the clamping end in the 3D model according to the coordinate positions of the ladle sprue and the long sprue installation part, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and calculate the registration transformation matrix from the long sprue installation part to the ladle sprue according to the coordinate positions of the ladle sprue and the long sprue installation part, and use the transformation matrix to align the point cloud data of the long sprue installation part and the ladle sprue. The model building unit is used to transform the registered point cloud data as a whole and build a 3D model of the long nozzle and clamping end.

6. The eye-in-the-hand vision positioning system for installing long nozzles on steel ladles according to claim 5, characterized in that, The processor is also used to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference using a path planning algorithm, and to calculate the path from the centroid position of the clamping end of the long nozzle installation part to the target position of the outlet using a path planning algorithm, and to send the planned path from the gripping surface of the clamping end to the gripped surface of the long nozzle, and the planned path from the long nozzle installation part to the outlet of the ladle to the mechanism control system.

7. The eye-to-hand manipulator vision positioning system for installing long nozzles on steel ladles according to claim 5, characterized in that, The visual acquisition device is mounted on the clamping unit using an eye-to-hand method.

8. The eye-to-hand manipulator vision positioning system for installing long nozzles on steel ladles according to claim 5, characterized in that, Visual acquisition equipment includes 3D structured light cameras.

9. The eye-to-hand manipulator vision positioning system for installing long nozzles on steel ladles according to claim 5, characterized in that, The clamping unit includes a robotic arm and a clamping end integrated with the robotic arm.

Images