A 3D reconstruction method and system for a scrap yard

Abstract

The present invention provides a three-dimensional reconstruction method and system for a scrap yard, including: when the operating equipment in the scrap yard is in a stationary state, obtaining a static three-dimensional model of the scrap pile at the corresponding position of the operating equipment; when the operating equipment is in a moving state, obtaining three-dimensional point cloud data of the scrap pile within the field of view of the operating equipment, obtaining the corresponding real-time pose according to the three-dimensional point cloud data, and stitching the three-dimensional point cloud data according to the real-time pose to obtain a dynamic three-dimensional model of the scrap yard; guiding the operating equipment to execute operating tasks according to the static three-dimensional model and the dynamic three-dimensional model, where the operating tasks include: scrap loading and unloading and / or moving towards a target pile; the present invention performs three-dimensional environment modeling for different states of the operating equipment, so as to execute operating tasks more accurately and efficiently, and improve accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent steel manufacturing, and particularly to a three-dimensional reconstruction method and system for a scrap yard. Background Art

[0002] In today's world with rapid technological development, manual labor and ordinary power machines are still the main labor forces at many construction sites. As the main operating vehicle for handling scrap steel in the steelmaking process, the steel grab needs to operate almost around the clock in harsh environments such as dust and sand. In this context, in order to improve the production efficiency of the steel mill and reduce the heavy labor of workers' three-shift system, the unmanned operation of the steel grab in the yard operation will become a breakthrough point to change the traditional operation mode. The unmanned operation of the steel grab means that by deploying multiple sensors on the body and robotic arm of the steel grab, three-dimensional point cloud and color information of obstacles, target stack shapes, dump trucks, etc. around the operation environment are obtained, and functions such as environmental obstacle detection, three-dimensional reconstruction of stack shapes, detection of grasping areas, and pose estimation and trailer recognition of dump trucks are realized, and finally the automatic grasping of scrap steel piles is achieved. Among them, the three-dimensional reconstruction of the steel material stack type and the acquisition of geometric information lay the foundation for the next step of recognition and determination of the area to be grasped, and are a key step to achieve the goal of unmanned operation of the steel grab.

[0003] Currently, there are mainly two types of three-dimensional reconstruction methods: image-based and point cloud-based. Image-based three-dimensional reconstruction is to restore the two-dimensional information of an image into a three-dimensional model, but its data acquisition and modeling process are complex and not suitable for application in the steel grabbing process; point cloud-based three-dimensional reconstruction is to convert three-dimensional data into point cloud data and then process it into a three-dimensional model, which has the characteristics of fast speed and high accuracy. In order to meet the application scenarios such as automatic movement and grasping of the steel grab in the yard operation, it is urgent to make further improvements in terms of reconstruction real-time performance and integrity. Summary of the Invention

[0004] In view of the above problems existing in the prior art, the present invention proposes a three-dimensional reconstruction method and system for a scrap yard, mainly solving the problem that the work such as scrap steel loading and unloading in the existing scrap yard depends on manual labor and has low efficiency.

[0005] In order to achieve the above object and other objects, the technical solution adopted by the present invention is as follows.

[0006] A three-dimensional reconstruction method for a scrap yard includes:

[0007] When the operating equipment in the scrap yard is in a stationary state, obtain a static three-dimensional model of the scrap steel pile at the corresponding position of the operating equipment;

[0008] When the working device is in a moving state, obtain the three-dimensional point cloud data of the scrap steel pile within the field of view of the working device, obtain the corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain the dynamic three-dimensional model of the scrap steel yard;

[0009] Guide the working device to execute the operation task according to the static three-dimensional model and the dynamic three-dimensional model, where the operation task includes: scrap steel loading and unloading and / or moving towards the target pile.

[0010] Optionally, obtaining the static three-dimensional model and the dynamic three-dimensional model includes:

[0011] The working device collects three-dimensional point cloud data and preprocesses the three-dimensional point cloud data, where the preprocessing includes: filtering the three-dimensional point cloud data using a filtering algorithm;

[0012] Input the preprocessed three-dimensional point cloud data into the pose estimation model, obtain the real-time pose corresponding to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose;

[0013] Remove the ground data from the spliced three-dimensional point cloud data, perform clustering operations on the remaining three-dimensional point cloud data using a clustering algorithm to obtain the three-dimensional point cloud data of the scrap steel pile, and represent the point cloud data of the scrap steel pile using an octree to obtain the static three-dimensional model or the dynamic three-dimensional model.

[0014] Optionally, the pose estimation model includes: the LeGO-LOAM model.

[0015] Optionally, the filtering algorithm includes a cylindrical filtering algorithm and a voxel filtering algorithm.

[0016] Optionally, removing the ground data from the spliced three-dimensional point cloud data includes:

[0017] Use the random sample consensus algorithm to fit the point cloud data corresponding to the ground, and remove the point cloud data corresponding to the ground from the three-dimensional point cloud data.

[0018] Optionally, the working device collects three-dimensional point cloud data, including:

[0019] Set a multi-line sensor on the working device, where the multi-line sensor is composed of multiple single-line sensors, and obtain three-dimensional point cloud data through the multi-line sensor;

[0020] Select the three-dimensional point cloud data obtained by the single-line sensor for pose estimation, obtain the real-time pose corresponding to the single-line sensor, and calculate the real-time poses of the remaining single-line sensors in the multi-line sensors according to the relative positions between the multi-line sensors and the real-time pose of the single-line sensor;

[0021] Convert the real-time poses of the single-line sensors to the world coordinate system, and splice the point cloud data under different real-time poses.

[0022] Optionally, after obtaining the static three-dimensional model or the dynamic three-dimensional model, perform a closing operation on the static three-dimensional model or the dynamic three-dimensional model to compensate for the missing parts on the surface of the corresponding three-dimensional model.

[0023] A three-dimensional reconstruction system for a scrap yard, comprising:

[0024] A static model acquisition module, configured to obtain a static three-dimensional model of the scrap pile at the corresponding position of the working equipment when the working equipment in the scrap yard is in a stationary state;

[0025] A dynamic model acquisition module, configured to obtain the three-dimensional point cloud data of the scrap pile within the field of view of the working equipment when the working equipment is in a moving state, obtain the corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain the dynamic three-dimensional model of the scrap yard;

[0026] A job navigation module, configured to guide the working equipment to execute job tasks according to the static three-dimensional model and the dynamic three-dimensional model, where the job tasks include: scrap loading and unloading and / or moving towards a target pile.

[0027] As described above, the three-dimensional reconstruction method and system for a scrap yard of the present invention have the following beneficial effects.

[0028] By performing three-dimensional modeling on the surrounding environment of the working equipment in different states, accurately understanding the distribution of the piles is beneficial for accurately positioning the working equipment and using complex working environments, reducing manual participation, and improving working efficiency and working accuracy. Brief Description of the Drawings

[0029] Figure 1 It is a schematic flowchart of the three-dimensional reconstruction method for a scrap yard in an embodiment of the present invention.

[0030] Figure 2 It is a flowchart of the random sample consensus algorithm.

[0031] Figure 3 It is a schematic flowchart of the pose estimation model. Detailed Embodiments

[0032] The following uses specific concrete examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0033] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present invention. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0034] Please refer to Figure 1 , the present invention provides a three-dimensional reconstruction method for a scrap yard. The method includes the following steps.

[0035] Step S01, when the operating equipment in the scrap yard is in a stationary state, obtain the static three-dimensional model of the scrap pile at the corresponding position of the operating equipment;

[0036] Step S02, when the operating equipment is in a moving state, obtain the three-dimensional point cloud data of the scrap pile within the field of view of the operating equipment, obtain the corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain the dynamic three-dimensional model of the scrap yard;

[0037] Step S03, guide the operating equipment to execute an operation task according to the static three-dimensional model and the dynamic three-dimensional model, where the operation task includes: scrap loading and unloading and / or moving towards a target pile.

[0038] In an embodiment, the operating equipment may include a steel grabber. A lidar sensor is arranged on the body of the steel grabber. Specifically, a 16-line radar sensor can be used to collect the three-dimensional point cloud data around the steel grabber. Since the influence of the relative movement between the steel grabber and the surrounding environment on collecting the three-dimensional point cloud data is small in the static state, more detailed point cloud data can be obtained. In the static state, the operating equipment needs to perform tasks such as steel grabbing and waste loading and unloading, and more precise three-dimensional modeling of the pile, etc., is required to accurately guide operating equipment such as steel grabbers to complete the corresponding steel grabbing tasks according to the modeling. In the dynamic state, the requirements for pile shape recognition for obstacle avoidance or travel route planning are relatively low. Therefore, the operating state of the operating equipment can be divided into a stationary state and a moving state. The corresponding static three-dimensional model and dynamic three-dimensional model are reconstructed respectively for the stationary state and the moving state.

[0039] In one embodiment, obtaining the static three-dimensional model and the dynamic three-dimensional model includes:

[0040] The working device collects three-dimensional point cloud data and preprocesses the three-dimensional point cloud data, where the preprocessing includes: filtering the three-dimensional point cloud data using a filtering algorithm;

[0041] Input the preprocessed three-dimensional point cloud data into a pose estimation model to obtain the real-time pose corresponding to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose;

[0042] Remove the ground data from the spliced three-dimensional point cloud data, perform clustering operations on the remaining three-dimensional point cloud data using a clustering algorithm to obtain the three-dimensional point cloud data of the scrap steel pile, and represent the point cloud data of the scrap steel pile using an octree to obtain the static three-dimensional model or the dynamic three-dimensional model.

[0043] Specifically, taking the steel grabber as an example, when the steel grabber is stationary, three-dimensional reconstruction and representation of the material pile are performed. First, two point cloud filtering algorithms, cylindrical filtering and voxel filtering, are used to preprocess the point cloud data. The specific preprocessing process is not elaborated here.

[0044] After the preprocessing is completed, the ground can be segmented and removed using the Random Sample Consensus (RANSAC) algorithm. Please refer to Figure 2 , the steps of random sample consensus include:

[0045] Step 1, randomly select three points;

[0046] Step 2, construct a plane according to the three selected points. The plane can be expressed as Ax + By + Cz + D = 0. Substitute the three points into the plane equation to obtain the plane function representation;

[0047] Step 3, the ground area range can be used as the plane range threshold, and the points within the range threshold are included in a point set.

[0048] Step 4, determine whether the number of points in the point set is greater than the set threshold. If it is greater, end the sampling. If it is not greater, reselect a subset with better fitting effect for re-update and iteration, and randomly select three points from the subset to perform point set sampling according to the previous steps.

[0049] After removing the ground point cloud data, the point cloud data of the material pile is distinguished from the vehicle, people, and material pile through the DBSCAN density clustering algorithm. Assume that the data set corresponding to the input point cloud data is D = (x1, x2,..., xm). Then the specific density description of DBSCAN is defined as follows:

[0050] ∈-neighborhood (i.e., the third neighborhood): For xj ∈ D, its ∈-neighborhood contains the set of neighborhood point data in dataset D whose distance from xj is no greater than ∈, that is, N∈(xj) = {xi ∈ D|distance(xi, xj) ≤ ∈}, and the number of this set of neighborhood point data is denoted as |N∈(xj)|

[0051] Core object: For any sample xj ∈ D, if the N∈(xj) corresponding to its ∈-neighborhood contains at least MinPts samples, that is, if |N∈(xj)| ≥ MinPts, then xj is a core object (i.e., core point data).

[0052] Density direct reach: If xi is located in the ∈-neighborhood of xj and xj is a core object, then it is said that xi is density directly reachable from xj. Note that the reverse is not necessarily true, that is, at this time, it cannot be said that xj is density directly reachable from xi, unless xi is also a core object.

[0053] Density reachable: For xi and xj, if there exists a sample sequence p1, p2,..., pT, satisfying p1 = xi, pT = xj, and pT+1 is density directly reachable from pT, then it is said that xj is density reachable from xi. That is to say, density reachability satisfies transitivity. At this time, the transfer samples p1, p2,..., pT-1 in the sequence are all core objects, because only core objects can make other samples density directly reachable. Note that density reachability also does not satisfy symmetry, which can be obtained from the asymmetry of density direct reach.

[0054] Density connected: For xi and xj, if there exists a core object sample xk such that both xi and xj are density reachable from xk, then it is said that xi and xj are density connected. Note that the density connected relationship satisfies symmetry. The specific process of clustering will not be elaborated here. Finally, the octree method is used to reconstruct and represent the stockpile surface.

[0055] In one embodiment, when the steel grab is in an active state, the LeGO-LOAM algorithm framework can be used to perform real-time pose estimation on the moving steel grab, and the algorithm framework is as Figure 3 shown. And the acquired point cloud data is processed, and through frame-by-frame stitching and mapping, real-time pose estimation and three-dimensional scene reconstruction based on a single radar are realized.

[0056] In one embodiment, the operating device collects three-dimensional point cloud data, including:

[0057] A multi-line sensor is set on the operating device, where the multi-line sensor is composed of multiple single-direction sensors, and three-dimensional point cloud data is obtained through the multi-line sensor;

[0058] Select the three-dimensional point cloud data obtained by the single-line sensor for pose estimation, obtain the real-time pose corresponding to the single-line sensor, and calculate the real-time poses of the remaining single-line sensors in the multi-line sensors according to the relative positions between the multi-line sensors and the real-time pose of the single-line sensor;

[0059] Convert the real-time poses of each single-line sensor to the world coordinate system, and splice the point cloud data under different real-time poses.

[0060] Specifically, based on the real-time pose estimation of a single radar, combined with the extraction of the stockpile point cloud, the scenes of the grab crane and the stockpile in motion under the condition of multiple radars are reconstructed. The generated global scene can assist the intelligent operation of the grab crane, such as selecting the grabbing point and obtaining the yard distribution. The specific method is as follows: continue to use LeGO-LOAM to calculate the poses of other radars relative to the existing 16-line radar according to the converted existing 16-line radar, perform real-time estimation, and then convert all pose information to the world coordinate system to splice the point cloud data. Then, the spliced point cloud data is input into the function of removing the ground and clustering, and then the stockpile point cloud is extracted for octree representation. Finally, the reconstruction and representation of the global stockpile under the condition of multiple radars are realized. On this basis, the closing operation method of digital image processing is added to compensate the reconstructed model and improve its surface.

[0061] According to the position information of the scrap steel stockpile in the static three-dimensional model, guide the manipulator of the grab crane to adjust its pose and move above the stockpile to complete the grabbing of waste or unloading the waste in the hopper. According to the position information of the stockpile in the dynamic three-dimensional model, guide the grab crane to select one of the stockpiles as the target stockpile. During the movement of the grab crane towards the target stockpile, plan the movement path according to the positions of other stockpiles and accurately avoid obstacles.

[0062] In this embodiment, a three-dimensional reconstruction system for a scrap steel yard is also provided, which is used to execute the three-dimensional reconstruction method of the scrap steel yard described in the foregoing method embodiment. Since the technical principle of the system embodiment is similar to that of the foregoing method embodiment, the same technical details will not be repeated.

[0063] In one embodiment, the three-dimensional reconstruction system of the scrap steel yard includes:

[0064] A static model acquisition module, which is used to obtain the static three-dimensional model of the scrap steel stockpile at the corresponding position of the working equipment when the working equipment in the scrap steel yard is in a stationary state;

[0065] A dynamic model acquisition module, configured to obtain three-dimensional point cloud data of a scrap steel pile within the field of view of the working device when the working device is in a moving state, acquire a corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain a dynamic three-dimensional model of the scrap steel yard;

[0066] A job navigation module, configured to guide the working device to execute job tasks according to the static three-dimensional model and the dynamic three-dimensional model, where the job tasks include: scrap steel loading and unloading and / or moving towards a target pile.

[0067] In summary, the present invention provides a three-dimensional reconstruction method and system for a scrap steel yard, which realizes real-time three-dimensional reconstruction of a pile under multiple radar conditions in actual situations such as when a steel grab is stationary and moving, with clear and complete effects, provides a basis for further determining the area to be grabbed by the steel grab, and ultimately realizes unmanned scrap steel transportation. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0068] The above embodiments are only illustrative of the principles and effects of the present invention, and are not used to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A three-dimensional reconstruction method for a scrap yard, characterized in that, it includes: When the operating equipment in the scrap yard is in a stationary state, obtain the static three-dimensional model of the scrap heap at the corresponding position of the operating equipment; When the operating equipment is in a moving state, obtain the three-dimensional point cloud data of the scrap heap within the field of view of the operating equipment, obtain the corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain the dynamic three-dimensional model of the scrap yard; Obtain the static three-dimensional model and the dynamic three-dimensional model, including: the operating equipment collects three-dimensional point cloud data and preprocesses the three-dimensional point cloud data, where the preprocessing includes: filtering the three-dimensional point cloud data using a filtering algorithm; inputting the preprocessed three-dimensional point cloud data into a pose estimation model, obtaining the real-time pose corresponding to the three-dimensional point cloud data, and splicing the three-dimensional point cloud data according to the real-time pose; removing the ground data in the spliced three-dimensional point cloud data, performing a clustering operation on the remaining three-dimensional point cloud data using a clustering algorithm to obtain the three-dimensional point cloud data of the scrap heap, and representing the point cloud data of the scrap heap using an octree to obtain the static three-dimensional model or the dynamic three-dimensional model; Guide the operating equipment to execute operation tasks according to the static three-dimensional model and the dynamic three-dimensional model, where the operation tasks include: scrap loading and unloading and / or moving towards the target heap.

2. The three-dimensional reconstruction method for a scrap yard according to claim 1, characterized in that, the pose estimation model includes: the LeGO-LOAM model.

3. The three-dimensional reconstruction method for a scrap yard according to claim 1, characterized in that, the filtering algorithm includes a cylindrical filtering algorithm and a voxel filtering algorithm.

4. The three-dimensional reconstruction method for a scrap yard according to claim 1, characterized in that, removing the ground data in the spliced three-dimensional point cloud data includes: fitting the point cloud data corresponding to the ground using a random sample consensus algorithm, and removing the point cloud data corresponding to the ground from the three-dimensional point cloud data.

5. The three-dimensional reconstruction method for a scrap yard according to claim 1, characterized in that, the operating equipment collects three-dimensional point cloud data, including: setting a multi-line sensor on the operating equipment, where the multi-line sensor is composed of multiple single-line sensors, and obtaining three-dimensional point cloud data through the multi-line sensor; selecting the three-dimensional point cloud data obtained by a single-line sensor for pose estimation, obtaining the real-time pose corresponding to the single-line sensor, and calculating the real-time poses of the remaining single-line sensors in the multi-line sensor according to the real-time pose of the single-line sensor and the relative positions between the multi-line sensors; converting the real-time poses of each single-line sensor to the world coordinate system, and splicing the point cloud data under different real-time poses.

6. The three-dimensional reconstruction method for a scrap yard according to claim 1, characterized in that, after obtaining the static three-dimensional model or the dynamic three-dimensional model, perform a closing operation on the static three-dimensional model or the dynamic three-dimensional model to compensate for the missing parts on the surface of the corresponding three-dimensional model.

7. A three-dimensional reconstruction system for a scrap yard, characterized in that, it includes: A static model acquisition module, which is used to obtain the static three-dimensional model of the scrap heap at the corresponding position of the working equipment when the working equipment in the scrap yard is in a stationary state; A dynamic model acquisition module, which is used to obtain the three-dimensional point cloud data of the scrap heap within the field of view of the working equipment when the working equipment is in a moving state, obtain the corresponding real-time pose according to the three-dimensional point cloud data, and splice the three-dimensional point cloud data according to the real-time pose to obtain the dynamic three-dimensional model of the scrap yard; Obtaining the static three-dimensional model and the dynamic three-dimensional model includes: the working equipment collects three-dimensional point cloud data and preprocesses the three-dimensional point cloud data, where the preprocessing includes: filtering the three-dimensional point cloud data using a filtering algorithm; inputting the preprocessed three-dimensional point cloud data into a pose estimation model to obtain the real-time pose corresponding to the three-dimensional point cloud data, and splicing the three-dimensional point cloud data according to the real-time pose; removing the ground data in the spliced three-dimensional point cloud data, performing a clustering operation on the remaining three-dimensional point cloud data using a clustering algorithm to obtain the three-dimensional point cloud data of the scrap heap, and representing the point cloud data of the scrap heap using an octree to obtain the static three-dimensional model or the dynamic three-dimensional model; A job navigation module, which is used to guide the working equipment to execute job tasks according to the static three-dimensional model and the dynamic three-dimensional model, where the job tasks include: scrap loading and unloading and / or moving towards the target heap.

Images