Eye-in-hand manipulator visual positioning system and method for mounting long nozzle on steel ladle

Through the "eye-in-hand" visual positioning method, three-dimensional point cloud data and path planning algorithm are used to solve the positioning accuracy and path adaptation problems of industrial robots in long water outlet replacement, and efficient and safe automatic replacement of long water outlets is achieved.

CN118989911AActive Publication Date: 2024-11-22北京瓦特曼智能科技有限公司

Patent Information

Application Number
CN202411410259.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-11-22
Estimated Expiration
2044-10-10

AI Technical Summary

Technical Problem

During the replacement of long water ports, existing industrial robots have problems such as low positioning accuracy of vision sensors, limited fixed viewing angle and insufficient adaptive ability to path planning, resulting in insufficient operating efficiency and safety.

Method used

Using the 'eye-in-hand' visual positioning method, by collecting and preprocessing three-dimensional point cloud data, extracting features and building a three-dimensional model, calculating the center coordinate difference of the grab surface, using the path planning algorithm to calculate the robotic arm path, and combining with the 3D structured optical camera to achieve Dynamic real-time perception and path adjustment.

Benefits of technology

It improves the accuracy and flexibility of long water outlet replacement, reduces manual intervention, improves automation level and production efficiency, and reduces operating risks and costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118989911A_ABST
    Figure CN118989911A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of visual positioning, and particularly relates to an eye-in-hand manipulator visual positioning system and method for mounting a long nozzle on a steel ladle, and the method comprises the steps: collecting and preprocessing three-dimensional point cloud data of the long nozzle and a clamping end; extracting features of the long nozzle and the clamping end from the point cloud data, registering the features with pre-stored template features, and constructing a three-dimensional model; extracting center coordinates of a grabbed surface of the long nozzle in the three-dimensional model and a grabbed surface of the clamping end, calculating a difference and planning a path; calculating a registration transformation matrix from a long nozzle mounting part to a steel ladle collector nozzle, and aligning the point cloud data and planning a path by using the transformation matrix; and sending the planned path to a control system. Through an eye-in-hand visual sensing mode, the positioning precision and the real-time adaptive capacity of path planning are improved, the problem of insufficient path planning is solved, the position change of the turret can be effectively handled, the path deviation and the docking failure risk are reduced, and the automation degree and the production efficiency are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual positioning, and in particular, relates to a visual positioning system and method for an eye-in-hand manipulator for installing a long shroud on a ladle. Background Art

[0002] In the process of iron and steel metallurgy, ladle pouring and shroud replacement in the continuous casting workshop are key operating links. The smooth progress of this process directly affects the quality of molten steel and production efficiency. Traditional shroud replacement usually relies on manual operation, but this method has problems such as low operating accuracy, low efficiency, and great safety hazards. In order to solve these problems and improve operating efficiency and safety, many steel mills have begun to use industrial manipulators to automatically replace shrouds.

[0003] Existing industrial manipulators automatically complete the grasping, positioning and installation of long nozzles through preset programs and visual perception systems, replacing manual hydraulic grippers and realizing unmanned operations. These manipulators mainly rely on visual sensors and automatic control systems to complete tasks. However, the existing technology still faces some technical challenges and limitations in the application process, and needs further improvement and optimization.

[0004] Specifically, the current visual sensing technology of industrial manipulators in the replacement of long nozzles has the following problems:

[0005] 1. Limitations of visual sensors: Most existing visual systems use binocular cameras and line laser technology. Although binocular cameras are low-cost and can work under a variety of lighting conditions, their positioning accuracy is poor in long-distance or low-contrast scenes, and they cannot obtain accurate depth information for surfaces with less texture, affecting the overall positioning accuracy. At the same time, the positioning algorithm of the binocular camera is relatively complex and consumes a lot of computing resources. Although line laser technology has the advantages of high precision and high resolution, it can only obtain depth data of a single line, and it is difficult to directly obtain complete three-dimensional information. It is necessary to obtain global data through the movement of sensors or targets, which not only increases the complexity of operation, but also increases the cost of equipment.

[0006] 2. "Eyes outside" visual sensor fixing method: Most visual sensors are fixed on the outside of the manipulator. Although this "eyes outside" method has better stability, it cannot adjust the view angle with the movement of the manipulator due to the fixed view angle, thus limiting the flexibility of operation. When tasks at different angles or positions need to be processed, the fixed view angle may not provide sufficient information support.

[0007] 3. Insufficient path planning and real-time adaptive capabilities: Existing manipulators usually use fixed path planning methods and lack the ability to adapt to dynamic changes. For example, when the rotation and lifting position of the large bag turntable changes, the manipulator cannot adjust the path in real time to adapt to the new position. In this case, path deviation or docking failure is prone to occur, and recalibration before operation or reliance on manual intervention is often required, which significantly reduces the system's degree of automation and operating efficiency.

[0008] Therefore, steel mills urgently need an improved method that can solve the above problems, such as using the "eye-in-hand" visual positioning method to improve the operational flexibility and accuracy of the robot during the replacement of the ladle shroud. Summary of the invention

[0009] The object of the present invention is to provide a visual positioning system and method for an eye-in-hand robot for installing a long shroud on a ladle, so as to solve the problems of visual sensor limitations, fixed viewing angle limitations, and insufficient path planning and real-time adaptive capabilities existing in the above-mentioned background technology.

[0010] To achieve the above-mentioned purpose, on the one hand, the present invention provides the following technical solutions: a method for visually positioning an eye-in-hand manipulator for installing a long shroud on a ladle, which is used to control a manipulator having a camera device on a mechanical clamping end, and the manipulator is used to clamp the long shroud docked with the ladle lower shroud, and the method comprises:

[0011] Step 1: Collecting three-dimensional point cloud data of the long nozzle and the clamping end, and preprocessing the three-dimensional point cloud data;

[0012] Step 2: Extract the features of the shroud and the clamping end from the preprocessed point cloud data, align the extracted features with the respective pre-stored template features, transform the aligned point cloud data as a whole and construct a three-dimensional model of the shroud and the clamping end;

[0013] Step 3: Extract the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end in the three-dimensional model, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference;

[0014] Step 4: According to the coordinate positions of the ladle outlet and the shroud mounting part, the registration transformation matrix from the shroud mounting part to the ladle outlet is calculated, the point cloud data of the shroud mounting part and the ladle outlet are aligned using the transformation matrix, and the path planning algorithm is used to calculate the path of the clamping end from the centroid position of the shroud mounting part to the target position of the outlet;

[0015] Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long nozzle, and the planned path from the long nozzle installation part to the ladle outlet to the mechanism control system.

[0016] Furthermore, in step 2, the three-dimensional model is constructed based on the aligned feature points of the long nozzle and the clamping end, and the transformation matrix between the long nozzle and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the long nozzle are calculated respectively. The transformation matrix is ​​used to align the long nozzle point cloud and the clamping end point cloud to the template coordinate system respectively, and merge them into a complete three-dimensional model.

[0017] Furthermore, in step 3, the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end are obtained by a centroid calculation method.

[0018] Furthermore, the centroid calculation method in step 3 includes:

[0019] Sum the coordinates of each feature point in the feature point set of the captured surface, add up the coordinates of all feature points and divide by the total number of feature points to obtain the centroid coordinates of the captured surface of the long nozzle:

[0020]

[0021] Sum the coordinates of each feature point in the feature point set of the gripping surface of the clamping end, add the coordinates of all feature points and divide by the total number of feature points to obtain the centroid coordinates of the gripping surface of the clamping end:

[0022]

[0023] Where: Indicates the centroid coordinates of the captured surface of the long shroud; represents the centroid coordinates of the gripping surface of the clamping end; n represents the total number of matched feature point pairs; p i represents the i-th feature point in the long shroud point cloud; q i Indicates the i-th feature point that matches the pre-stored template of the long nozzle.

[0024] Furthermore, in step 4, the alignment feature of the long nozzle mounting part is the normal vector of the mounting surface, and the alignment feature of the ladle outlet is the normal vector of the outlet. By calculating the rotation matrix and translation vector between the normal vectors, a homogeneous transformation matrix from the long nozzle mounting part to the ladle outlet is formed.

[0025] On the other hand, the present invention also provides an eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle, wherein the eye-in-hand manipulator visual positioning system is used to implement an eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle, comprising:

[0026] Visual acquisition equipment, used to obtain 3D point cloud data of the shroud and the clamping end;

[0027] The clamping unit is used to clamp the shroud and perform operations of grabbing, moving and docking the shroud with the ladle outlet;

[0028] A processor is used to pre-process the acquired three-dimensional point cloud data of the shroud and the clamping end, extract the features of the shroud and the clamping end from the pre-processed point cloud data, and align them with the respective pre-stored template features, determine the transformation relationship between the extracted features and the template features, extract the center coordinates of the grasped surface of the shroud and the grasping surface of the clamping end in the three-dimensional model according to the coordinate positions of the ladle shroud and the shroud mounting part, calculate the difference between the center coordinates of the grasped surface and the grasping surface, and calculate the registration transformation matrix from the shroud mounting part to the ladle shroud according to the coordinate positions of the ladle shroud and the shroud mounting part, and align the point cloud data of the shroud mounting part and the ladle shroud using the transformation matrix;

[0029] The model building unit is used for overall transformation of the registered point cloud data and building a three-dimensional model of the long nozzle and the clamping end.

[0030] Furthermore, the processor is also used to calculate the path of the clamping end from the current position to the grasped surface of the long water spout based on the coordinate difference using a path planning algorithm, and to calculate the path of the clamping end from the center of mass position of the long water spout mounting part to the target position of the water spout using a path planning algorithm, and send the planned path from the grasping surface of the clamping end to the grasped surface of the long water spout, and the planned path from the long water spout mounting part to the water spout of the ladle to the mechanism control system.

[0031] Furthermore, the visual acquisition device is installed on the clamping unit in an eye-in-hand manner.

[0032] Furthermore, the visual acquisition device includes a 3D structured light camera.

[0033] Furthermore, the clamping unit includes a mechanical arm and a clamping end integrated with the mechanical arm.

[0034] The beneficial effects of the present invention are:

[0035] First, the eye-in-hand manipulator visual positioning method for installing the long nozzle of the ladle of the present invention realizes the high-precision automatic replacement of the long nozzle through intelligent path planning and adaptive control. Using three-dimensional point cloud data collection and feature extraction, the system calculates and adjusts the robot arm path in real time to ensure accurate grasping and installation, and adapt to dynamic environmental changes. This method significantly improves the operation accuracy, success rate and flexibility, reduces manual intervention, and improves the level of automation, thereby optimizing the quality of molten steel and production efficiency, reducing operational risks and costs, and providing an efficient and safe solution for steel metallurgical production.

[0036] Second, in the preferred implementation, the present invention can accurately align the point cloud data to the template coordinate system and construct a complete three-dimensional model by calculating and applying the transformation matrix between the long nozzle and the clamping end and the pre-stored template, as well as the transformation matrix between the clamping end and the long nozzle. This precise alignment method enhances the consistency and stability of the long nozzle and the clamping end during the grasping and installation process, and reduces deviations and error accumulation during operation.

[0037] Third, in the preferred implementation, the present invention adopts the centroid calculation method to obtain the center coordinates of the long nozzle and the grasping surface of the clamping end, and realizes the alignment operation of the long nozzle installation part and the ladle lower nozzle through normal vector calculation, and combines the calculation of the rotation matrix and the translation vector to accurately plan the path and posture adjustment of the clamping end. This method ensures the accuracy of grasping and docking during the installation of the long nozzle, reduces the deviation in path planning, and improves the adaptability and operating efficiency of the manipulator under complex working conditions.

[0038] Fourth, the eye-in-hand manipulator visual positioning system for installing the long water nozzle of the ladle of the present invention integrates the visual acquisition equipment, the clamping unit, the processor and the model building unit, and coordinates the work of each module through a unified path planning algorithm to ensure the efficient operation of the whole process from data acquisition, feature extraction, model building to path execution. The mechanical arm in the clamping unit works in coordination with the clamping end, and combined with the processor's execution of path planning instructions, it realizes the precise docking operation from the long water nozzle gripping surface to the target position of the ladle water nozzle. This highly integrated design reduces the complexity of the intermediate links, improves the system's automation level and operating efficiency, and reduces operating costs and the need for manual intervention.

[0039] Fifth, in the preferred implementation, the visual acquisition device of the present invention adopts a 3D structured light camera and is installed on the clamping unit in an eye-in-hand manner, thereby realizing dynamic real-time three-dimensional perception of the long water outlet and the clamping end. This method greatly improves the accuracy of the long water outlet grasping, moving and docking, and at the same time allows the system to quickly adjust the operation path according to real-time environmental changes, thereby enhancing the flexibility and adaptability of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle according to the present invention;

[0041] Figure 2 It is a three-dimensional structural diagram of the ladle pouring unmanned manipulator with eyes in the hand of the present invention.

[0042] Among them, 1-visual acquisition equipment; 2-clamping unit; 20-clamping end; 21-mechanical arm. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0044] The terms such as up, down, left, right, front and back in this application document are based on the positional relationships shown in the drawings. If the drawings are different, the corresponding positional relationships may also change accordingly, so they cannot be understood as limiting the scope of protection.

[0045] In this application, the terms "install", "connected", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, or mutual communication, a direct connection, or an indirect connection through an intermediate medium, the internal connection of two components, or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0046] As the instruction manual Figure 1 The present invention records a visual positioning method of an eye-in-hand manipulator for installing a long shroud on a ladle, comprising:

[0047] Step 1: Collect three-dimensional point cloud data of the long nozzle and the clamping end, and pre-process the three-dimensional point cloud data.

[0048] It should be noted that the shroud is placed on the shroud shelf in the designated area. During the system initialization or planning stage, the shroud placement area coordinates will be pre-set and stored in the robot's control system. The shroud placement area coordinates can be derived from the process layout diagram or manual measurement to ensure that their accuracy meets the operational requirements. The system reads the preset coordinates of the shroud placement area from the storage and sets the read coordinates as the target position for the robot to move. The robot uses its own path planning module (such as A*, Dijkstra or other path planning algorithms) to calculate the optimal path from the current position to the preset coordinates, and moves to the shroud placement area along the planned path.

[0049] Specifically, the long nozzle in the placement area is photographed using a visual acquisition device to obtain the three-dimensional point cloud data of the long nozzle. The top of the long nozzle is provided with a gripped surface and a mounting portion, the gripped surface is used to cooperate with the gripping surface of the clamping end, and the mounting portion is used to connect the drain. The three-dimensional point cloud data includes the point cloud data of the gripped surface and the mounting portion. The visual acquisition device is installed on the robot arm in an "eye-in-hand" manner, so while acquiring the three-dimensional point cloud data of the long nozzle, the three-dimensional point cloud data of the clamping end is also acquired synchronously.

[0050] In the embodiment of the present application, the collected 3D point cloud data of the shroud and the clamping end are preprocessed using the wavelet denoising method. Taking the 3D point cloud data preprocessing of the shroud as an example:

[0051] Firstly, the 3D point cloud data of the long shroud is transformed by wavelet transform to decompose the data into signal components of different frequencies. Wavelet decomposition can analyze point cloud data at different spatial frequencies to help identify the global morphology and local detail features of the data. The decomposed signals include low-frequency components and high-frequency components. The low-frequency components reflect the main morphology and global structure of the long shroud, and the high-frequency components contain details, local changes and noise information. Through wavelet transform, the noise and useful features in the point cloud data can be separated, providing a basis for subsequent noise removal.

[0052] Secondly, threshold processing is performed on the high-frequency components (including details and noise) obtained by decomposition, and an appropriate threshold is set (through experience, statistical methods or adaptive algorithms) to screen the signal amplitude in the high-frequency components. The noise components below the threshold are set to zero or reduced, and the components above the threshold (important details) are retained, thereby removing noise and reducing random errors and irregular fluctuations in point cloud data.

[0053] Finally, the low-frequency and high-frequency components that have been threshold processed are recombined, and the frequency components are synthesized into complete point cloud data through inverse wavelet transform. During the reorganization process, the original spatial characteristics of the long shroud point cloud data are accurately restored to ensure that the denoised data retains the original geometric structure and positional relationship, thereby generating denoised point cloud data.

[0054] Step 2: Extract the features of the shroud and the clamping end from the preprocessed point cloud data, align the extracted features with the respective pre-stored template features, transform the aligned point cloud data as a whole and construct a three-dimensional model of the shroud and the clamping end.

[0055] Step 2 specifically includes:

[0056] Step 2.1: Perform feature extraction on the denoised point cloud data of the shroud and the clamping end respectively.

[0057] The features extracted from the shroud include edge features, hole features, gripped surface features, and mounting features. The edge features are to extract the edges and corner areas of the shroud surface and define the overall contour of the shroud. The hole features are to identify the locations of the mounting holes of the shroud, which are important reference points for alignment. The gripped surface refers to the surface on the shroud that is specially designed to cooperate with the clamping end. These features usually include planes, recessed areas, or other structures that are easy to clamp. The mounting features extract the mounting planes, mounting holes, and mounting edges, which are the core parts for cooperation with the clamping end.

[0058] The features extracted from the clamping end include the gripping surface features and the clamping position features. The gripping surface features are used to identify the edges and contact points of the gripping surface of the clamping end, which serve as alignment references during the clamping process. The clamping position features are used to extract the geometric features of the clamping position, including the specific position of the gripping surface and the features of the fixture surface.

[0059] In the embodiment of the present application, the SIFT feature extraction algorithm (Scale-Invariant Feature Transform) is used to identify the geometric feature points of the shroud and the clamping end. Take the feature extraction of the shroud as an example:

[0060] First, by using Gaussian filters, the point cloud data is blurred at different levels to construct images of multiple scales, so that the feature points at different scales can be analyzed. This is equivalent to observing the image with a "blurring mirror" of different intensities, so as to identify the feature points that are still obvious at various scales.

[0061] Then, in these different levels of images, we look for "bright spots" or "dark spots" that are more prominent than the surrounding points. These are the geometric feature points of interest. These feature points will remain unchanged at different scales and directions. We also create an "identity tag" for each feature point, which captures the image change trend around the feature point (similar to the direction of the surrounding light and dark changes), which can better describe the nature of the feature point.

[0062] Finally, the extracted feature points are screened to remove those with low contrast, not obvious enough, and those located at the edge and prone to errors. This helps to reduce noise and unstable feature points. The screened feature point set retains the main geometric information of the long nozzle, such as the edge, the changing position of the surface, the unique features of the grasped surface and the installation part. These high-quality feature points provide a reliable data basis for subsequent matching and registration.

[0063] Step 2.2: Use the nearest neighbor search to preliminarily align the feature points of the shroud and the clamping end with their respective pre-stored template feature points, and preliminarily align the feature points of the clamping end with the shroud. Then use the ICP (iterative closest point) algorithm to completely overlap the features of the preliminarily aligned shroud and clamping end with the template features, and completely overlap the feature points of the clamping end with the shroud.

[0064] Specifically, the nearest neighbor search is to find the pairing of each point with the closest point in the template by comparing the Euclidean distance between feature points to achieve approximate alignment. For each feature point, the minimum distance to the template feature point set is calculated. The point pairs with the smallest distance are regarded as matching pairs, and these pairs are marked as the basis for preliminary alignment. Based on the preliminary matching point pairs, a preliminary rigid body transformation matrix is ​​calculated, including translation and rotation, and applied to the point cloud data of the long nozzle and the clamping end to make them preliminarily aligned with the template point cloud.

[0065] The ICP (Iterative Closest Point) algorithm performs fine registration of the feature points of the shroud and the template feature points on the basis of preliminary alignment. Through multiple iterations, the ICP algorithm continuously minimizes the error between the matching point pairs, adjusts the point cloud posture and position of the shroud and the clamping end, and achieves the precise docking of the shroud point cloud data and the pre-stored template of the shroud, the precise docking of the clamping end point cloud data and the pre-stored template of the clamping end, and the precise docking of the matching point cloud of the clamping end and the shroud.

[0066] Step 2.3: Based on the registered feature points of the shroud and the clamping end, calculate the transformation matrix between the shroud and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the shroud, respectively. Use the transformation matrix to align the shroud point cloud and the clamping end point cloud to the template coordinate system as a whole, and align the grasping position of the clamping end to the grasped surface of the shroud. Merge the point clouds of the shroud and the clamping end, respectively, and construct a three-dimensional model of the shroud and the clamping end using the mesh generation method.

[0067] In the embodiment of the present application, the transformation matrix of the long nozzle and its pre-stored template is taken as an example to illustrate the calculation process of the transformation matrix:

[0068] S1: Calculate the centroid of the long shroud point cloud and the template point cloud for the matched feature point pairs respectively.

[0069] The centroid of the shroud is expressed by the following formula:

[0070]

[0071] The centroid of the pre-stored template of the long nozzle is expressed by the following formula:

[0072]

[0073] Where: n represents the total number of matched feature point pairs; p i represents the i-th feature point in the long shroud point cloud; q i Indicates the long nozzle pre-stored template and p i The i-th feature point that matches.

[0074] S2: Subtract the corresponding centroid from each feature point in the long nozzle and the long nozzle pre-stored template to obtain the point cloud coordinates after removing the centroid.

[0075] The point cloud of the long shroud without centroid is expressed as:

[0076] The point cloud expression of the pre-stored template of the long nozzle without the centroid is:

[0077] S3: Calculate the covariance matrix H of the point cloud without centroid of the long nozzle and template point cloud, perform singular value decomposition (SVD) on the covariance matrix H, and calculate the rotation matrix R using the result of the singular value decomposition.

[0078] S5: Using the rotation matrix R and the centroid position of the point cloud, the translation vector T is calculated, and the centroid of the measured point cloud of the long shroud is translated to the centroid position of the pre-stored template point cloud of the long shroud.

[0079] S6: Combine the rotation matrix R and the translation vector T to construct a 4x4 homogeneous transformation matrix to obtain the transformation matrix between the long nozzle and its pre-stored template.

[0080] Step 3: Extract the center coordinates of the grasped surface of the long nozzle and the grasping surface of the clamping end in the three-dimensional model, calculate the difference between the center coordinates of the grasped surface and the grasping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the grasped surface of the long nozzle based on the coordinate difference.

[0081] Assume that the feature point set of the captured surface is {p1,p2,……,p n}, where each feature point p i With coordinates (x i ,y i ,z i ). Using the centroid calculation method, calculate the centroid of the feature points of the grasped surface, sum the coordinates of all feature points respectively, and then divide by the total number of feature points n. The centroid calculation method of the grasped surface of the long nozzle is determined by the following formula:

[0082]

[0083] According to formula (3), the center coordinate p of the grasped surface is obtained install =(x instart ,y instart ,z instart ).

[0084] Similarly, let the feature point set of the grasping surface be {q1,q2,……,q n}, where each feature point q i With coordinates (x j ,y j ,z j). Using the centroid calculation method, the coordinates of all the feature points of the gripping surface are summed up respectively, and then divided by the total number of feature points m. The centroid calculation method of the gripping surface of the clamping end is determined by the following formula:

[0085]

[0086] According to formula (4), the center coordinate q of the gripping surface of the clamping end is obtained start =(x start ,y start ,z start ).

[0087] The center coordinate p of the grasped surface install and the center coordinate q of the gripping surface of the clamping end start Used for path planning and docking operations at the gripping end.

[0088] Path planning algorithms include A*, Dijkstra, RRT*, etc. In the embodiment of the present application, taking the A* algorithm as an example, the path calculation process of the clamping end from the current position to the position of the gripped surface of the long nozzle includes:

[0089] S1: The workspace of the clamping end operation is modeled as a graph. The nodes in the graph represent the accessible positions and the edges represent the possible movement paths. Let the current position of the clamping end be the starting point node q, and the coordinates of point q be (x start ,y start ,z start ), assuming that the center position of the long nozzle grasped surface is the target node p, and the coordinates of point p are (x instart ,y instart ,z instart ), calculate the center coordinate difference between the gripping position of the clamping end and the gripped surface of the long nozzle.

[0090] The coordinate difference between the gripping position of the clamping end and the gripped surface of the long nozzle is:

[0091] Δp=p install -q start =[x instart -x start ,y instart -y start ,z instart -z start ]

[0092] S2: Create an initialization structure, which includes an open list, a closed list, a g table, and an f table. Select the node with the smallest f value as the current node, and update the costs of adjacent nodes until the target node p is found.

[0093] It should be noted that the total estimated cost of each node is f = g + h, where h is the heuristic estimated cost, which is provided by the coordinate difference Δp. The initial f value of the starting point q is equal to its heuristic estimated cost, and the initial f values ​​of other nodes are infinite.

[0094] S3: Starting from the target node p, trace the path backward through the predecessor node table until returning to the starting point q, and arrange the nodes in the tracing order to form the shortest path from the starting point to the target point.

[0095] Through these steps, the A* algorithm can help the clamping end efficiently find and execute the optimal path from the current position to the gripped surface of the long nozzle, ensuring the accuracy of the clamping and installation operations.

[0096] Step 4: According to the coordinate positions of the ladle outlet and the shroud mounting part, calculate the registration transformation matrix from the shroud mounting part to the ladle outlet, use the transformation matrix to align the point cloud data of the shroud mounting part and the ladle outlet, and use the path planning algorithm to calculate the path of the clamping end from the center of mass position of the shroud mounting part to the target position of the outlet.

[0097] It should be noted that the actual measured position coordinates of the ladle outlet have been calculated before executing step 1 of this application. The calculation method and the coordinate calculation method of the long outlet installation part can refer to the calculation principle of steps 1-2 of this application and will not be repeated here.

[0098] Specifically, step 4 includes:

[0099] Step 4.1: When the centroid position of the gripping surface of the clamping end has been aligned with the centroid position of the gripped surface of the shroud, obtain the centroid position coordinates of the shroud mounting part and the measured position coordinates of the ladle outlet, and calculate the translation vector from the shroud mounting part to the ladle outlet.

[0100] The centroid position coordinate c of the shroud installation part install =(x instart ,y instart ,z instart ), the measured position coordinates of the ladle outlet p tundish =(x tundish ,y tundish ,z tundish ), the translation vector from the shroud installation to the ladle outlet is:

[0101] T align =p tundish -c install [x tundish -x instart ,y tundish -y instart ,z tundish -z instart

[0102] Step 4.2: Extract the alignment features of the shroud mounting part and the ladle outlet, and calculate the rotation matrix between the two.

[0103] Specifically, extract the normal vector n of the installation surface from the three-dimensional point cloud of the long nozzle installation part install and extract the normal vector n of the ladle outlet from the 3D point cloud or pre-stored template of the ladle outlet tundish , calculate the rotation matrix using the Rodriguez formula:

[0104]

[0105] Where: θ represents the normal vector n install and n tundish The angle between them (i.e. the amount that the mounting part of the shroud needs to be rotated to align with the ladle outlet), and u represents the rotation axis.

[0106] The rotation axis u is the normal vector n install and the normal vector n tundish The product of install and n tundish are parallel, then u is a zero vector and no rotation is required.

[0107] Step 4.3: Combine the translation vector and the rotation matrix to form the final homogeneous transformation matrix.

[0108] The homogeneous transformation matrix is:

[0109]

[0110] Step 4.4: Set the current position of the clamping end as the starting point of the path planning, and the position of the ladle outlet as the target position of the path planning. The path planning algorithm calculates the path from the starting point to the target point.

[0111] In an embodiment of the present application, step 4.4 may adopt the same A* planning algorithm as step 3, and the specific process will not be repeated here.

[0112] Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long nozzle, and the planned path from the long nozzle installation part to the ladle outlet to the mechanism control system.

[0113] Specifically, the path point sets of the grasping path and the docking path are formatted into standard data packets (such as JSON, XML, or a custom binary format) to ensure that the data packets have good compatibility and transmission stability. Clear identifiers (such as "grasping path data" and "docking path data") are added to the data packets to ensure that the control system can correctly identify the path type and purpose. The mechanism control system receives the path data packet, parses the data packet content, and extracts the path point set and related instructions. The control system commands the gripping end to perform grasping and docking operations according to the order and instructions of the path point set.

[0114] Refer to the instruction manual Figure 2 The present invention also records a visual positioning system of an eye-in-hand manipulator for installing a long shroud on a ladle. The eye-in-hand manipulator visual positioning system is used to implement a visual positioning method of an eye-in-hand manipulator for installing a long shroud on a ladle, and includes a visual acquisition device 1, a clamping unit 2, a processor and a model building unit.

[0115] The visual acquisition device 1 is installed on the clamping unit 2 in an eye-in-hand manner and is used to obtain three-dimensional point cloud data of the long water outlet and the clamping end.

[0116] In the embodiment of the present application, the visual acquisition device is installed above the clamping unit 2 in an "eye-in-hand" manner. The visual acquisition device includes a binocular camera, a line laser device, a 3D structured light camera, etc. The present invention preferably uses a 3D structured light camera, which can quickly and accurately obtain three-dimensional data of an object by projecting a specific pattern and capturing a deformed image, combined with high-precision calculation and real-time processing technology.

[0117] The clamping unit 2 includes a robotic arm 21 and a clamping end 20 integrated with the robotic arm 21. The clamping end 20 has a clamping surface that matches the clamped surface of the long water nozzle. The clamping unit 2 is used to clamp the long water nozzle and perform operations of grabbing, moving and docking the long water nozzle with the ladle outlet.

[0118] The processor is used to pre-process the 3D point cloud data of the shroud and the clamping end, extract the features of the shroud and the clamping end from the pre-processed point cloud data, and align them with the respective pre-stored template features to determine the transformation relationship between the extracted features and the template features, extract the center coordinates of the grasped surface of the shroud and the grasping surface of the clamping end in the 3D model according to the coordinate positions of the ladle shroud and the shroud installation part, calculate the difference between the center coordinates of the grasped surface and the grasping surface, and calculate the center coordinates of the clamping end based on the coordinate difference using a path planning algorithm. The path from the current position to the grasped surface of the shroud, and according to the coordinate positions of the ladle outlet and the shroud mounting part, the registration transformation matrix from the shroud mounting part to the ladle outlet is calculated, the point cloud data of the shroud mounting part and the ladle outlet are aligned using the transformation matrix, and the path planning algorithm is used to calculate the path of the clamping end from the center of mass position of the shroud mounting part to the target position of the outlet, and the planned path from the grasping surface of the clamping end to the grasped surface of the shroud, and the planned path from the shroud mounting part to the ladle outlet are sent to the mechanism control system.

[0119] The model building unit is used for overall transformation of the registered point cloud data and building a three-dimensional model of the long nozzle and the clamping end.

[0120] The present invention realizes high-precision automatic replacement of the ladle installation nozzle through intelligent path planning and adaptive control. It uses three-dimensional point cloud data and centroid calculation method for precise alignment and path planning to improve the accuracy and stability of grasping and installation. It integrates visual acquisition equipment, clamping unit and processor, and uses visual acquisition equipment to perceive and dynamically adjust the operation path in real time, improve the automation level, operation efficiency and system flexibility, optimize molten steel quality and reduce costs.

[0121] The above is only an embodiment of the present invention, and the common sense such as the known specific structure and characteristics in the scheme is not described in detail here. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claims involved.

Claims

1. A method for visually positioning an eye-in-hand manipulator for installing a long shroud on a ladle, which is used to control a manipulator with a camera device on a mechanical clamping end, and the manipulator is used to clamp the long shroud docked with the ladle lower shroud, characterized in that: Methods include: Step 1: Collecting three-dimensional point cloud data of the long nozzle and the clamping end, and preprocessing the three-dimensional point cloud data; Step 2: Extract the features of the shroud and the clamping end from the preprocessed point cloud data, align the extracted features with the respective pre-stored template features, transform the aligned point cloud data as a whole and construct a three-dimensional model of the shroud and the clamping end; Step 3: Extract the center coordinates of the gripped surface of the long nozzle and the gripping surface of the clamping end in the three-dimensional model, calculate the difference between the center coordinates of the gripped surface and the gripping surface, and use the path planning algorithm to calculate the path from the current position of the clamping end to the gripped surface of the long nozzle based on the coordinate difference; Step 4: According to the coordinate positions of the ladle outlet and the shroud mounting part, the registration transformation matrix from the shroud mounting part to the ladle outlet is calculated, the point cloud data of the shroud mounting part and the ladle outlet are aligned using the transformation matrix, and the path planning algorithm is used to calculate the path of the clamping end from the centroid position of the shroud mounting part to the target position of the outlet; Step 5: Send the planned path from the gripping surface of the clamping end to the gripped surface of the long nozzle, and the planned path from the long nozzle installation part to the ladle outlet to the mechanism control system.

2. The eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle according to claim 1 is characterized in that: In step 2, the three-dimensional model is constructed based on the aligned feature points of the long nozzle and the clamping end, and the transformation matrix between the long nozzle and its pre-stored template, the transformation matrix between the clamping end and its pre-stored template, and the transformation matrix between the clamping end and the long nozzle are calculated respectively. The transformation matrix is ​​used to align the long nozzle point cloud and the clamping end point cloud to the template coordinate system respectively, and merge them into a complete three-dimensional model.

3. The eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle according to claim 1 is characterized in that: In step 3, the center coordinates of the grasped surface of the long nozzle and the grasping surface of the clamping end are obtained by the centroid calculation method.

4. The eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle according to claim 3 is characterized in that: The centroid calculation method in step 3 includes: Sum the coordinates of each feature point in the feature point set of the captured surface, add up the coordinates of all feature points and divide by the total number of feature points to obtain the centroid coordinates of the captured surface of the long nozzle: Sum the coordinates of each feature point in the feature point set of the gripping surface of the clamping end, add the coordinates of all feature points and divide by the total number of feature points to obtain the centroid coordinates of the gripping surface of the clamping end: Where: Indicates the centroid coordinates of the gripped surface of the long shroud; represents the centroid coordinates of the gripping surface of the clamping end; n represents the total number of matched feature point pairs; p i represents the i-th feature point in the long shroud point cloud; q i Indicates the i-th feature point that matches the pre-stored template of the long nozzle.

5. The eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle according to claim 1 is characterized in that: In step 4, the alignment feature of the long nozzle mounting part is the normal vector of the mounting surface, and the alignment feature of the ladle outlet is the normal vector of the outlet. By calculating the rotation matrix and translation vector between the normal vectors, a homogeneous transformation matrix from the long nozzle mounting part to the ladle outlet is formed.

6. An eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle, characterized in that: The eye-in-hand manipulator visual positioning system is used to implement the eye-in-hand manipulator visual positioning method for installing a long shroud on a ladle, comprising: Visual acquisition equipment, used to obtain 3D point cloud data of the shroud and the clamping end; The clamping unit is used to clamp the shroud and perform operations of grabbing, moving and docking the shroud with the ladle outlet; A processor is used to pre-process the acquired three-dimensional point cloud data of the shroud and the clamping end, extract the features of the shroud and the clamping end from the pre-processed point cloud data, and align them with the respective pre-stored template features, determine the transformation relationship between the extracted features and the template features, extract the center coordinates of the grasped surface of the shroud and the grasping surface of the clamping end in the three-dimensional model according to the coordinate positions of the ladle shroud and the shroud mounting part, calculate the difference between the center coordinates of the grasped surface and the grasping surface, and calculate the registration transformation matrix from the shroud mounting part to the ladle shroud according to the coordinate positions of the ladle shroud and the shroud mounting part, and align the point cloud data of the shroud mounting part and the ladle shroud using the transformation matrix; The model building unit is used for overall transformation of the registered point cloud data and building a three-dimensional model of the long nozzle and the clamping end.

7. The eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle according to claim 6 is characterized in that: The processor is also used to calculate the path of the clamping end from the current position to the grasped surface of the long nozzle based on the coordinate difference using a path planning algorithm, and to calculate the path of the clamping end from the center of mass position of the long nozzle mounting part to the target position of the outlet nozzle using a path planning algorithm, and to send the planned path from the grasping surface of the clamping end to the grasped surface of the long nozzle, and the planned path from the long nozzle mounting part to the outlet nozzle of the ladle to the mechanism control system.

8. The eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle according to claim 6, characterized in that: The visual acquisition device is installed on the clamping unit in an eye-in-hand manner.

9. The eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle according to claim 6, characterized in that: Visual acquisition equipment includes 3D structured light cameras.

10. The eye-in-hand manipulator visual positioning system for installing a long shroud on a ladle according to claim 6, characterized in that: The clamping unit includes a mechanical arm and a clamping end integrated with the mechanical arm.

Citation Information

Patent Citations

  • Mechanical arm sorting and grabbing system and method based on 3D vision

    CN111151463A

  • Method and device for automatically searching steel-clad opening and readable medium thereof

    CN115147481A

  • Grabbing method and system applied to mobile phone widget production line

    CN116128871A

  • Uncalibrated mechanical arm grabbing method and device based on hand-eye camera

    CN116690575A

  • Mechanical arm intelligent finishing treatment system for dead heads of large castings

    CN117464668A

Cited By

  • Visual tactile perception-based attitude estimation method and system for mechanical handheld acupuncture needle

    CN120070579A