Bolt position identification method and device and bolt identification system

Abstract

The invention provides a bolt position identification method and device and a bolt identification system. The method comprises the steps that point cloud data of a current mine card are acquired, and the point cloud data are obtained based on scanning of a laser sensor; template point cloud data is obtained, and the template point cloud data is data of a template of a pre-configured bolt; and performing global matching on the point cloud data and the template point cloud data to obtain the position of the bolt. According to the scheme, the problem of low efficiency of manually identifying the position of the bolt of the mine card in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of digital maintenance of electric wheels of large-tonnage mining trucks, and in particular to a method and device for identifying bolt positions, a computer program product and a bolt identification system. Background Art

[0002] In the process of modern mining and transportation, large-tonnage mining trucks (mining trucks) are key equipment and undertake the task of efficient transportation of ore. The electric wheels of mining trucks are an important part of them, which directly affects the operation stability and safety of mining trucks. With the rapid development of intelligent manufacturing and automation technology, how to improve the assembly accuracy and maintenance efficiency of mining truck electric wheels has become an important issue in the industry.

[0003] Traditional maintenance of electric wheels on mining trucks mainly relies on manual operation, especially for the detection and position identification of bolts. In actual operation, there are a large number of bolts, which are often distributed in complex structures. Manual identification of the positions of these bolts is not only time-consuming and labor-intensive, but also prone to errors, especially in harsh mining environments where workers face multiple unfavorable conditions such as high temperature, high dust, and high noise. These factors make it difficult to ensure the efficiency and accuracy of manual maintenance, which may lead to extended maintenance time for mining trucks and even cause safety accidents. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, computer program product and bolt identification system for identifying the position of a bolt, so as to at least solve the problem of low efficiency in manually identifying the position of the bolts of a mining truck in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for identifying the position of a bolt is provided, comprising: obtaining point cloud data of a current mining truck, wherein the point cloud data is obtained based on scanning by a laser sensor; obtaining template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template; and globally matching the point cloud data and the template point cloud data to obtain the position of the bolt.

[0006] Optionally, obtaining the point cloud data of the current mining truck includes: obtaining initial point cloud data, wherein the initial point cloud data is initial data obtained by scanning with the laser sensor; filtering the initial point cloud data to obtain intermediate point cloud data; and using the RANSAC algorithm to segment the intermediate point cloud data to obtain the point cloud data.

[0007] Optionally, the RANSAC algorithm is used to segment the intermediate point cloud data to obtain the point cloud data, including: obtaining the Euclidean distance between the first point cloud and the second point cloud, wherein the first point cloud is any point cloud in the intermediate point cloud data, and the second point cloud is any point cloud in the intermediate point cloud data except the first point cloud; when the Euclidean distance is less than or equal to a preset distance threshold and the total number of point clouds is within a preset range, the first point cloud and the second point cloud are divided into a cluster, wherein the total number of point clouds is the sum of the number of the first point cloud and the number of the second point cloud.

[0008] Optionally, obtaining the template point cloud data includes: obtaining a CAD model file, wherein the CAD model file is a pre-stored drawing file of the structure of the bolt; converting the CAD model file into an STL format to obtain an STL format file; obtaining three-dimensional geometric information of the STL format file, and performing triangulated meshing processing to obtain a network model file; using a uniform sampling algorithm to collect a target number of points from the network model file to obtain the template point cloud data.

[0009] Optionally, the point cloud data and the template point cloud data are globally matched to obtain the position of the bolt, including: calculating the similarity between the point cloud data and the template point cloud data; extracting the point in the point cloud data with the highest similarity to the template point cloud data to obtain the matching point; obtaining the position of the matching point in the point cloud data to obtain the position of the bolt.

[0010] Optionally, before acquiring the point cloud data of the current mining truck, the method further includes: calibrating the laser sensor; and scanning the mining truck using the calibrated laser sensor.

[0011] Optionally, after globally matching the point cloud data and the template point cloud data to obtain the position of the bolt, the method further includes: converting the position of the bolt into a target coordinate system, wherein the target coordinate system is a robot coordinate system; in the target coordinate system, controlling the robot to disassemble and / or tighten the bolt.

[0012] According to another aspect of the present application, a device for identifying the position of a bolt is provided, comprising: a first acquisition unit, used to acquire point cloud data of a current mining truck, wherein the point cloud data is obtained based on scanning by a laser sensor; a second acquisition unit, used to acquire template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template; and an identification unit, used to globally match the point cloud data with the template point cloud data to obtain the position of the bolt.

[0013] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the bolt position identification methods are implemented.

[0014] According to another aspect of the present application, a bolt identification system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the bolt position identification methods.

[0015] By applying the technical solution of the present application, the position of the bolts can be identified by performing global template matching on the extracted point cloud data of the scene and the generated template point cloud data. Manual identification of the position of the bolts is no longer required, which shortens the identification time, reduces the operational risks of workers, achieves high-precision detection of the bolts, and improves the efficiency of bolt position identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for identifying a bolt position provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a method for identifying a bolt position provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram showing a flow chart of another method for identifying a bolt position of the present application;

[0020] Figure 4 A structural block diagram of a bolt position identification device provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, the efficiency of manually identifying the position of the bolts of the mining truck in the prior art is low. To solve the above problem, the embodiments of the present application provide a method, device, computer program product and bolt identification system for identifying the position of the bolts.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for identifying a bolt position according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a method for identifying the position of a bolt running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 2 FIG. 1 is a flow chart of a method for identifying a bolt position according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, obtaining point cloud data of the current mining truck, wherein the point cloud data is obtained based on laser sensor scanning;

[0033] Specifically, the laser sensor is used to scan the target area on the electric wheel of the mining truck to obtain detailed three-dimensional data. The laser sensor emits a laser beam and receives the laser reflected from the surface of the object to accurately measure the distance between each point on the surface of the object and the sensor, thereby constructing three-dimensional point cloud data on the surface of the object. These data contain the three-dimensional position information of the bolts on the electric wheel of the mining truck, which is the basis for subsequent bolt position identification.

[0034] Step S202, obtaining template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template;

[0035] Specifically, a template point cloud data of a bolt is prepared in advance, which is usually generated based on the CAD model of the bolt. The template point cloud data contains the standard geometric features of the bolt, such as shape, size and position, so that it can be compared and matched with the actual point cloud data later.

[0036] Step S203: globally matching the point cloud data with the template point cloud data to obtain the position of the bolt.

[0037] Specifically, the point cloud data actually acquired is globally matched with the template point cloud data, that is, the best geometric alignment between the point clouds is found. The precise position of the bolt on the electric wheel of the mining truck can be determined through the matching results.

[0038] The specific implementation of global matching may include multiple sub-steps, such as the extraction of feature descriptors (such as FPFH, SPFH, etc.), matching of feature points, and optimization algorithms based on rigid body transformation (such as ICP algorithm) to refine the matching results and ensure the high accuracy of the bolt position. When matching, considering that the actual point cloud data may be affected by noise, occlusion or missing, the algorithm should have the ability to handle these situations to improve the robustness and accuracy of matching.

[0039] Through this embodiment, the position of the bolt is identified by performing global template matching on the extracted point cloud data of the scene and the generated template point cloud data. Manual identification of the position of the bolt is no longer required, which shortens the identification time, reduces the operation risk of workers, achieves high-precision detection of the bolts, and improves the efficiency of bolt position identification.

[0040] Specifically, in recent years, with the rapid development of artificial intelligence and machine vision technology, automated detection methods based on machine vision have gradually become important tools in the industrial field. Machine vision technology acquires images or videos through cameras, and uses image processing and pattern recognition algorithms for data analysis, which can efficiently and accurately identify specific targets in complex environments. The application of this technology can not only improve detection efficiency, but also significantly reduce human errors. However, although machine vision has been successfully applied in industrial inspection, it still faces many challenges in the field of bolt position identification of electric wheels of large-tonnage mining trucks.

[0041] Therefore, there is an urgent need for a machine vision method for the digital maintenance system of electric wheels of large-tonnage mining trucks, focusing on the bolt position detection in high-precision robot assembly. It can not only significantly improve the assembly efficiency and accuracy, but also effectively reduce the risks of manual operation, and provide strong technical support for the intelligent and automated development of mines.

[0042] In the specific implementation process, the point cloud data of the current mining truck can be obtained through the following steps: obtaining initial point cloud data, wherein the initial point cloud data is the initial data obtained by scanning with the laser sensor; filtering the initial point cloud data to obtain intermediate point cloud data; using the RANSAC algorithm to segment the intermediate point cloud data to obtain the point cloud data.

[0043] In this solution, the original initial point cloud data can be obtained, and irrelevant information can be eliminated through filtering and segmentation processing to ensure better data quality, so that bolt position identification can be performed more efficiently and accurately in the future.

[0044] Specifically, the bolt scene point cloud data collected by the line laser sensor camera is preprocessed to extract the bolt target data, that is, the point cloud data.

[0045] Specifically, the bolt scene point cloud data of the workpiece photographed by the line laser vision sensor camera is subjected to through-filtering and outlier statistical filtering in sequence; the processed bolt scene point cloud data is segmented by the RANSAC (Random Sample Consensus) clustering algorithm, wherein the parameter values ​​of the RANSAC algorithm need to be adjusted until all the bolts can be correctly segmented.

[0046] Specifically, filtering processing refers to the straight-through filtering of the bolt scene point cloud data of the workpiece photographed by the line laser vision sensor camera. Assuming that the input point cloud is P = {p i |p i =(x i ,y i ,z i )}, the straight-through filtering formula is:

[0047] P filtered ={p i |T1≤p i ≤T2},

[0048] Among them, P filtered For the retained point cloud set, T1 and T2 are the filtering range thresholds, set T1 = -1600, T2 = -800;

[0049] For the processed point cloud P filtered , where p i Represents the coordinates of a point (x i ,y i ,z i ). For each point p i , calculate the average distance u to the k nearest neighbors in the neighborhood i :

[0050]

[0051] Among them, dist(p i ,p j ) is point p i To its neighbor point p j The Euclidean distance of

[0052] The outliers are judged by the threshold τ, and the standard deviation is σ:

[0053] τ=u+λ·σ,

[0054] Where u is all points u i The average value of , λ is the multiple coefficient for outlier determination, set λ = 1.0;

[0055] Keep the point cloud data P that meets the following conditions filtered2 :

[0056] P filtered2 ={p i ∈P filtered |u i ≤τ},

[0057] Among them, τ is the threshold for judging outliers, and point p i The average distance u i When it is less than or equal to τ, the point is retained.

[0058] Specifically, the initial point cloud data refers to the original three-dimensional data obtained directly from the bolt area of ​​the electric wheel of the mining truck by the laser sensor. These data usually contain a large number of points, each of which consists of its coordinates (XYZ) in three-dimensional space, and may also include additional attributes such as color or reflection intensity. Since it is obtained by direct scanning, the initial point cloud data often contains complex background information, such as other structures of the electric wheel, dust, noise points, etc. This information is not needed in the subsequent bolt detection, but may cause interference.

[0059] In actual operation, laser sensor scanning may be performed under different lighting conditions, angles and distances, which may lead to unevenness and quality changes in point cloud data. In order to obtain the most accurate initial point cloud data, it is necessary to calibrate the sensor, optimize the scanning settings, and keep the motorized wheel stable during scanning to avoid point cloud data errors caused by vibration or movement.

[0060] Specifically, filtering is an important step in point cloud preprocessing, which is used to remove noise, outliers or unnecessary background points to make subsequent bolt detection more accurate. Filtering methods can include straight-through filtering, statistical outlier filtering, neighborhood mean filtering, median filtering, etc. Each method has its own applicable scenarios and advantages.

[0061] In the specific implementation process, the filtering parameters can be adjusted according to the actual situation of the point cloud data, such as the range threshold of the straight-through filter or the standard deviation multiple of the statistical outlier filter. In addition, considering the particularity of the mining environment, a special filtering algorithm can be designed to deal with abnormal points caused by environmental factors such as dust and uneven lighting. The result of the filtering process is the intermediate point cloud data, which is cleaner and more focused on the target area, providing a better foundation for subsequent bolt detection.

[0062] Specifically, the RANSAC (Random Sampling Consensus) algorithm is a robust estimation method widely used in point cloud data processing, which is used to estimate the parameters of a mathematical model from a point set containing abnormal data. In this scheme, the RANSAC algorithm is used to segment the bolt target point cloud data from the intermediate point cloud data, that is, to accurately identify the point set of the bolt from the point cloud containing the bolt and other background.

[0063] Specifically, the implementation of the RANSAC algorithm usually involves the following steps: first, randomly select a portion of points as candidate model points; then, estimate the model parameters based on these points; then, based on the estimated model parameters, check which points in the point set are consistent with the model, these points are called inliers; finally, by iterating the above process, select the model parameters with the most inliers as the final model. In bolt detection, the goal of the RANSAC algorithm is to find a plane model or a specific geometric model that can best fit the point cloud data of the bolt. In order to improve the accuracy and efficiency of segmentation, it may be necessary to optimize the parameters of the RANSAC algorithm, such as the minimum sample set size, the maximum number of iterations, the consistency threshold, etc., to ensure that the bolt point set can be accurately segmented.

[0064] In the specific implementation process, the RANSAC algorithm is used to segment the above-mentioned intermediate point cloud data to obtain the above-mentioned point cloud data, which can be achieved through the following steps: obtaining the Euclidean distance between the first point cloud and the second point cloud, wherein the above-mentioned first point cloud is any point cloud in the above-mentioned intermediate point cloud data, and the above-mentioned second point cloud is any point cloud in the above-mentioned intermediate point cloud data except the above-mentioned first point cloud; when the above-mentioned Euclidean distance is less than or equal to the preset distance threshold and the total number of point clouds is within a preset range, the above-mentioned first point cloud and the above-mentioned second point cloud are divided into a cluster, wherein the above-mentioned total number of point clouds is the sum of the number of the above-mentioned first point cloud and the number of the above-mentioned second point cloud.

[0065] In this scheme, the bolt point clouds in the intermediate point cloud data can be effectively distinguished to form independent clusters to obtain point cloud data with better quality.

[0066] Specifically, the processed point cloud data P filtered2Perform RANSAC clustering, for any two points p i =(x i ,y i ,z i ) and p j =(x j ,y j ,z j ) between ij The calculation formula is:

[0067]

[0068] According to the above RANSAC clustering, for each point p i , calculate the distance between it and all other points, and determine whether it can be classified into a cluster, the conditions are as follows:

[0069] d ij ≤ClusterTolerance,

[0070] ClusterTolerance is the set distance threshold, set ClusterTolerance = 0.2;

[0071] According to the above RANSAC clustering, the clustering constraints are controlled by the minimum number of points MinClusterSize and the maximum number of points MaxClusterSize:

[0072] MinClusterSize ≤ ClusterPointsCount ≤ MaxClusterSize,

[0073] That is, a cluster is considered valid only when the number of points in the cluster satisfies the constraints between MinClusterSize and MaxClusterSize, where MinClusterSize = 20000 and MaxClusterSize = 30000.

[0074] Specifically, in the intermediate point cloud data, the spatial distance calculation process between any two point clouds (the first point cloud and the second point cloud). Euclidean distance is a measure of the straight-line distance between two points in three-dimensional space, which is obtained by calculating the square root of the sum of the squares of the differences between two points (or representative points of the point cloud) on the three coordinate axes of X, Y, and Z. If the distance between the first point cloud and the second point cloud is less than or equal to a given threshold (preset distance threshold), and the number of points after the two point clouds are merged is within a preset range, then the two point clouds will be considered to belong to the same cluster, that is, they may belong to the same bolt or the same component.

[0075] Specifically, the total number of point clouds within the preset range is to ensure that the number of points contained in each cluster is neither too few (possibly belonging to noise or partial structure) nor too many (possibly containing multiple bolts or part of a bolt and other structures). The setting of this range also needs to be based on the understanding of the size, shape and quality of the point cloud data of the bolt. For example, if the representation of a bolt in the point cloud requires about 5,000 points to describe, then the preset range may be set to 4,000 to 6,000 points to ensure the correctness of the clustering results.

[0076] Specifically, in a mining environment, due to the complex structure of the electric wheel and possible occlusion, the point cloud data obtained from a single scan may be incomplete, some bolts may be blocked or there may be gaps in the point cloud data. In order to accurately identify and segment the bolts, multi-view scanning and data fusion technology are used to ensure that the point cloud data of each bolt is complete enough while reducing the impact of occlusion.

[0077] In the specific implementation process, the template point cloud data can be obtained through the following steps: obtaining a CAD model file, wherein the above-mentioned CAD model file is a pre-stored drawing file of the structure of the above-mentioned bolt; converting the above-mentioned CAD model file into STL format to obtain an STL format file; obtaining the three-dimensional geometric information of the above-mentioned STL format file, and performing triangulated meshing processing to obtain a network model file; using a uniform sampling algorithm to collect a target number of points from the above-mentioned network model file to obtain the above-mentioned template point cloud data.

[0078] In this solution, by converting the CAD model into STL format, performing triangulated meshing and uniform sampling, the generated template point cloud data can more accurately reflect the true geometric shape of the bolt. The uniform sampling algorithm ensures the uniform distribution of the template point cloud data. Even if some areas of the bolt are obscured or damaged, it can still be matched based on the point cloud features of other complete areas, thereby improving the robustness of recognition.

[0079] Specifically, the bolt CAD model in .STL format can be triangulated to generate bolt model point cloud data.

[0080] Specifically, Solidworks software is used to convert the CAD model file stored in .SLDPRT format into a .STL model with a triangular mesh structure to ensure that the CAD model can be further processed and used; the bolt CAD model in .STL format is read to obtain its three-dimensional geometric information and perform triangular mesh processing on it; a uniform sampling algorithm is used to sample 10^5 points from the mesh model, and these points together constitute the point cloud data M of the bolt model. model as a matching template.

[0081] Specifically, a CAD (Computer-Aided Design) model file is a digital representation of a pre-designed and stored bolt structure. This file usually contains the precise geometry, size, and details of the bolt, and is the basis for creating template point cloud data. The STL (STereoLithography) format is a widely used 3D model file format that represents the surface of an object in the form of a triangular mesh. Converting a CAD model to the STL format can simplify the geometry and make it easier to be processed by a machine vision system.

[0082] Specifically, since the CAD model file has been triangulated to a certain extent when converted to the STL format, the triangulated meshing process here can be understood as further optimization of the STL file to ensure the consistency and quality of the mesh for subsequent point cloud data sampling.

[0083] Specifically, uniform sampling is to select points from the mesh model to build a point cloud that is evenly distributed in shape, which will serve as the template point cloud data for subsequent machine vision matching. The target number of point clouds should be sufficient to describe the geometric features of the bolt, and should not be too much to avoid increasing the computational complexity.

[0084] There are many uniform sampling algorithms, the most common ones are Poisson disk sampling, grid sampling, far point sampling, etc. The selection of a suitable algorithm should take into account the complexity of the bolt shape, the density of the point cloud data, and the limitation of computing resources. For example, Poisson disk sampling can generate a more uniformly distributed point cloud, but the computational cost is higher; while grid sampling is relatively simple and fast, but may lead to uneven point distribution in irregularly shaped areas.

[0085] In some embodiments, the point cloud data and the template point cloud data are globally matched to obtain the position of the bolt, which can be achieved through the following steps: calculating the similarity between the point cloud data and the template point cloud data; extracting the point in the point cloud data that has the highest similarity with the template point cloud data to obtain the matching point; obtaining the position of the matching point in the point cloud data to obtain the position of the bolt.

[0086] In this scheme, the point cloud data obtained in the scene will be compared with the template point cloud data of the bolt, and the similarity between the two will be calculated. The points with the highest similarity to the points in the template point cloud will be found from the scene point cloud. These points will be identified as matching points. The actual position of the bolt on the electric wheel can be determined through the position information of the matching points in the scene point cloud. This scheme can identify the position of the bolt on the electric wheel of the mining truck and achieve high-precision positioning.

[0087] Specifically, the scene point cloud data and the model point cloud data may be globally template matched to determine the position coordinates of the bolts.

[0088] Specifically, the above-mentioned determination of the position mainly includes the following steps:

[0089] Feature calculation, for the bolt template point cloud M model And the scene point cloud P filtered2 Calculate the FPFH (Fast Point Feature Histograms) feature descriptors, FPFHmodel and FPFHfiltered2, for each point respectively; FPFHmodel: This is the FPFH feature descriptor calculated from the bolt template point cloud data. The template point cloud data is extracted from the CAD model of the bolt and generated by triangulation and uniform sampling. The FPFHmodel feature descriptor is used to represent the standard geometric features of the bolt, and will be used to match the actual collected bolt scene point cloud data later. FPFHfiltered2: This is the FPFH feature descriptor calculated from the preprocessed bolt scene point cloud data. The scene point cloud data is converted from the bolt image actually taken on the electric wheel of a large-tonnage mining truck. The FPFHfiltered2 feature descriptor contains the geometric feature information of the actual bolt under different viewing angles and environments.

[0090] Similarity calculation, calculate the similarity between the FPFHmodel feature descriptor of each point in the bolt template point cloud and the FPFHfiltered2 feature descriptor of all points in the scene point cloud;

[0091] Select the point in the scene point cloud that has the highest similarity with the feature descriptor of the point in the bolt template point cloud as the matching point;

[0092] Repeat the above similarity calculation steps and matching steps until all points in the bolt template point cloud find their corresponding matching points in the scene point cloud;

[0093] Based on the matching point pairs obtained in the previous step, the rotation matrix and translation matrix between the bolt template point cloud and the scene point cloud are calculated. Specifically, suppose a point p in the bolt template point cloud is i and a point q in the scene point cloud i Correspondingly, the goal is to find a rotation matrix R s and a translation vector t s , by solving the minimization problem to align the transformed template point cloud with the scene point cloud as much as possible:

[0094]

[0095] Where N is the number of matching point pairs.

[0096] Specifically, feature calculation mainly includes the following contents:

[0097] Normal vector estimation, for a given bolt template point cloud M model And the scene point cloud P filtered2 , calculate the normal vector for each point respectively; the normal vector estimation is usually completed through the local neighborhood points of the point, and PCA (Principal Component Analysis) is used to estimate the normal direction of the local surface;

[0098] SPFH (Simplified Point Feature Histogram) calculation, for the scene point cloud P filtered2 Every point p in i , select a point p in its neighborhood j , calculate three angle features that describe the geometric relationship between point pairs;

[0099] α: point p i to p j The vector and point p i The angle between the normal vectors of

[0100] φ: point p j The normal vector of point p i The angle between the normal vectors of

[0101] θ: the angle between α and φ;

[0102] These three angle features are used to construct point p i The SPFH histogram:

[0103]

[0104] Among them, H(·) means that the angle values ​​are quantized and accumulated into the corresponding histogram;

[0105] Based on the SPFH calculation, the feature description is further enhanced by introducing weighted neighborhood information:

[0106]

[0107] Among them, ω ij It's point p i and p j The distance between them, k is the number of neighboring points.

[0108] Specifically, a feature descriptor is calculated for each point in the template point cloud data and the scene point cloud data, which can be FPFH (Fast Point Feature Histograms) or other descriptors based on geometric features. The calculation of the feature descriptor involves the local geometric environment of the point, such as its normal vector, curvature, and relationship with neighboring points.

[0109] Specifically, the feature descriptor of each point in the template point cloud data is compared with the feature descriptors of all points in the scene point cloud data to calculate the similarity. The similarity can be expressed by a distance metric between feature descriptors (such as Euclidean distance or cosine similarity). The smaller the distance, the higher the similarity.

[0110] Specifically, for each point in the template point cloud, find the point in the scene point cloud with the highest similarity and mark it as a matching point. In order to improve the robustness of the matching, it may be necessary to set a minimum similarity threshold. Only when the similarity is higher than this threshold, the matching point is confirmed.

[0111] Specifically, based on the matching points, the position of the bolt can be estimated. One approach is to use the least squares method or the iterative closest point (ICP) algorithm to optimize the relative position and pose between the matching points and the template point cloud. Through this process, a rotation and translation matrix can be obtained, which describes the best alignment between the bolt template point cloud and the scene point cloud.

[0112] Specifically, the position coordinates of the matching points in the scene point cloud are transformed into the robot coordinate system through the rotation and translation matrices obtained previously. This step is to achieve accurate positioning of the bolt position by transforming the point cloud coordinates from the sensor coordinate system to a coordinate system that the robot can understand.

[0113] Assume that the bolt template point cloud data consists of 10,000 points and the scene point cloud data contains 1 million points. First, the FPFH feature descriptor is calculated for all points, and then the similarity calculation is performed. For each point in the template, the similarity threshold is set to 0.05, which means that only when the distance between the feature descriptors is less than 0.05, it is considered a matching point. After the similarity calculation, about 10,000 matching points may be found, and the positions of these points in the scene point cloud data correspond to the actual positions of the bolts. Next, the alignment between the matching points and the template points is optimized through the ICP algorithm, and the precise position coordinates of the bolts on the electric wheel are obtained. Finally, these coordinates are converted into the coordinate system controlled by the robot, so that the robot can accurately locate and operate the bolts.

[0114] In some embodiments, before acquiring the point cloud data of the current mining truck, the method further includes the following steps: calibrating the laser sensor; and scanning the mining truck using the calibrated laser sensor.

[0115] In this solution, calibration can eliminate the hardware error of the sensor and ensure that the acquired point cloud data accurately corresponds to the real-world position. The calibration process ensures the accuracy of the data acquired by the sensor.

[0116] Specifically, a cylindrical calibration block can be used to calibrate the camera of the line laser vision sensor to obtain the internal and external parameters of the camera.

[0117] Specifically, multiple groups of images of cylindrical calibration blocks can be captured by a line laser vision sensor, where a cylindrical calibration block with known geometric dimensions is placed under the line laser vision sensor, and its images at different positions are obtained by changing the position of the calibration block; the known geometric features of the cylindrical calibration block and the specific point positions in the image are used to calculate the camera's intrinsic parameter matrix, including focal length, principal point position, and pixel scale factor. Correction eliminates the internal geometric errors caused by the lens to ensure that the image accurately reflects the real scene; based on the images of the cylindrical calibration block at different positions, the position and posture of the camera relative to the calibration block are calculated to obtain the camera's extrinsic parameter matrix, including the rotation matrix R c and the translation vector t c The rotation matrix represents the rotation posture of the camera. It describes the rotation relationship between the camera coordinate system and the world coordinate system. The direction, angle and rotation axis of the camera can be determined through the rotation matrix. The translation vector represents the position of the camera. It describes the position offset of the camera coordinate system in the world coordinate system. The position coordinates of the camera can be determined through the translation vector.

[0118] Specifically, calibration is a key step in the machine vision system, and its purpose is to determine the internal and external parameters of the laser sensor, including the focal length, principal point position, pixel scale factor and other internal parameters of the camera, as well as the rotation and translation of the camera relative to the world coordinate system. Through calibration, the hardware error of the sensor can be eliminated to ensure that the acquired point cloud data accurately corresponds to the real-world position. After the laser sensor is calibrated, it will be used to scan the mining truck and obtain the point cloud data of the mining truck surface. These point cloud data contain the three-dimensional geometric information of the mining truck surface, which will be used for subsequent bolt identification and position determination.

[0119] Specifically, suppose there is a line laser vision sensor used to scan bolts on a mining truck. During the calibration process, a cylindrical calibration block with known dimensions and features is used, placed in the field of view of the sensor, and multiple sets of photos are taken from different angles. Through these photos, the internal parameters of the sensor are calculated, such as the focal length f, the principal point position (cx, cy), and the pixel scale factors px, py. At the same time, the external parameters of the sensor relative to the calibration block are also obtained, namely the rotation matrix R and the translation vector T. After the calibration is completed, it can be ensured that the point cloud data obtained by the sensor can accurately reflect the positional relationship in the real world.

[0120] After the laser sensor is calibrated, it is installed on the robot arm of the mining truck maintenance robot to scan the electric wheel. The robot arm can adjust the position and angle of the sensor to ensure that all parts of the electric wheel can be scanned. During the scanning process, the sensor emits a line laser, and the camera captures the reflected light to form a series of three-dimensional coordinate information of points, thereby constructing the surface point cloud data of the electric wheel. Since the sensor has been calibrated, the position information of these point cloud data is accurate and reliable, and can be directly used for subsequent bolt identification and position determination.

[0121] In some embodiments, after globally matching the point cloud data and the template point cloud data to obtain the position of the bolt, the method further includes the following steps: converting the position of the bolt into a target coordinate system, wherein the target coordinate system is a robot coordinate system; in the target coordinate system, controlling the robot to disassemble and / or tighten the bolt.

[0122] In this solution, in the machine vision system, the point cloud data acquired by the sensor (such as a line laser vision sensor) is in the sensor's own coordinate system. However, the robot control system uses an independent coordinate system, usually called the robot coordinate system. In order for the robot to understand and locate the bolt, the position information of the bolt in the sensor coordinate system must be converted to the robot coordinate system. Once the position of the bolt is converted to the robot coordinate system, the robot control system can accurately control the robot's arm or tool based on these coordinate information to achieve automated removal or tightening of the bolt.

[0123] Specifically, the position of the bolt can be converted into a robot coordinate system controlled by the robot arm according to the calibration result, so as to perform subsequent disassembly or tightening operations.

[0124] Specifically, the conversion matrix between the camera coordinate system and the robot coordinate system can be obtained through the calibration results, including the rotation matrix R c and the translation vector t c ;

[0125] The obtained rotation matrix and translation matrix of the bolt in the scene based on the camera coordinate system are R s and t s ;

[0126] R s and t s The matrix is ​​transformed into the robot coordinate system. Assume that a point P in the camera coordinate system s The homogeneous coordinates of are expressed as [R s ·P+t s ], then the corresponding point P in the robot coordinate system r The homogeneous coordinates of are expressed as:

[0127] P r =R c ·(R s ·P+t s )+t c ,

[0128] Therefore, the transformed rotation matrix R r and the translation matrix t r It is expressed as:

[0129] R r =R c ·R s ,

[0130] t r =R c ·t s +t c ,

[0131] The R obtained by the above conversion r and t r As input, it is passed to the robot control system so that it can perform the corresponding task of removing or tightening the bolts.

[0132] Assume that the position and posture information of the bolts have been identified from the point cloud data of the electric wheel of the mining truck, which is obtained in the coordinate system of the online laser vision sensor. In order to enable the robot to perform the operation, the position of the bolts is first converted from the sensor coordinate system to the robot coordinate system using the rotation matrix determined during the calibration process.

[0133] Next, the robot control system will accurately control the robot arm to move to the bolt position based on the converted coordinate information and adjust the angle of the tool (such as a tightening gun or a disassembly tool) to adapt to the specific posture of the bolt. The robot can automatically select the appropriate operation mode (disassembly or tightening) and perform the operation according to the preset torque or force value to ensure that the tightening degree of the bolt meets the requirements.

[0134] In actual applications, the electric wheel of a mining truck may contain multiple bolts that need to be identified and operated. In order to optimize the robot operation process, a multi-target recognition strategy can be used to simultaneously identify the positions of multiple bolts and convert this information into the robot coordinate system. Then, the best operation sequence is planned based on the location of the bolts and the priority of the operation. For example, the robot may first operate those bolts that are easy to access and take a short operation time to reduce the overall operation time. When planning the operation path, the range of motion of the robot arm and obstacles should also be taken into account to avoid collisions or unnecessary movements.

[0135] During the robot's disassembly or tightening operation, the system should provide a real-time feedback mechanism. By scanning the operation position again, obtaining new point cloud data, and comparing it with the previous point cloud data, the effect of the operation can be evaluated. If it is found that the position or posture of the bolt after the operation is not as expected, the robot control system should be able to automatically adjust the operation strategy, such as repositioning or increasing the operation force, to ensure the success of the operation. In addition, real-time feedback can also be used to monitor abnormal conditions during the operation, such as bolt damage or jamming, so that timely measures can be taken.

[0136] In order to cope with different types of bolts and operating requirements, the robot should have the ability to adaptively adjust the operating mode. For example, for bolts with high tightness requirements, the robot can use high torque mode; while for loose and easily damaged bolts, low torque or low force mode should be used. Through integration with the machine vision system, the robot can automatically select the best operating mode according to the specific type and state of the bolt, which not only improves efficiency but also reduces potential risks in operation.

[0137] Specifically, in this solution, it can be a machine vision method for a digital maintenance system for electric wheels of large-tonnage mining trucks. The system includes: an industrial robot, a line laser vision sensor, a cylindrical calibration block and a robotic arm. The working robot of the system is used to disassemble or tighten the bolts in the electric wheels of mining trucks. The system is used to execute the above-mentioned bolt position identification scheme.

[0138] Specifically, in the above scheme, first, the line laser sensor camera is calibrated using a cylindrical calibration block; secondly, the bolt scene point cloud collected by the line laser sensor camera is preprocessed to extract the bolt target point cloud data; then, the .STL format CAD model of the bolt is triangulated to generate the bolt template point cloud data; then, the extracted scene point cloud data is globally template matched with the generated template point cloud data, and the bolt position coordinates are determined by the matching results; finally, the bolt position is converted to the robot coordinate system using the calibration result of the line laser vision sensor camera to achieve high-precision positioning and detection of the bolt.

[0139] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for identifying the bolt position of the present application will be described in detail below in conjunction with specific embodiments.

[0140] This embodiment relates to a specific method for identifying the position of a bolt. Figure 3 As shown, the following steps are included:

[0141] Step S1: calibrate the camera of the line laser vision sensor using a cylindrical calibration block to obtain the internal and external parameters of the camera;

[0142] Step S2: preprocessing the bolt scene point cloud data collected by the line laser sensor camera to extract bolt target data;

[0143] Step S3: triangulate the bolt CAD model in .STL format to generate point cloud data of the bolt model;

[0144] Step S4: Perform global template matching on the scene point cloud data in S2 and the model point cloud data in step S3 to determine the position coordinates of the bolts;

[0145] Step S5: According to the calibration result in S1, the position of the bolt is converted into the robot coordinate system controlled by the robot arm to facilitate subsequent disassembly or tightening operations.

[0146] The embodiment of the present application also provides a device for identifying the position of a bolt. It should be noted that the device for identifying the position of a bolt in the embodiment of the present application can be used to execute the method for identifying the position of a bolt provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0147] The following is an introduction to the bolt position identification device provided in the embodiment of the present application.

[0148] Figure 4 is a structural block diagram of a bolt position identification device according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0149] The first acquisition unit 10 is used to acquire point cloud data of the current mining truck, wherein the point cloud data is obtained based on scanning by a laser sensor;

[0150] The second acquisition unit 20 is used to acquire template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template;

[0151] The recognition unit 30 is used to globally match the point cloud data with the template point cloud data to obtain the position of the bolt.

[0152] Through this embodiment, the position of the bolt is identified by performing global template matching on the extracted point cloud data of the scene and the generated template point cloud data. Manual identification of the position of the bolt is no longer required, which shortens the identification time, reduces the operation risk of workers, achieves high-precision detection of the bolts, and improves the efficiency of bolt position identification.

[0153] During the specific implementation process, the first acquisition unit includes a first acquisition module, a filtering module and a segmentation module. The first acquisition module is used to acquire initial point cloud data, wherein the initial point cloud data is the initial data obtained by scanning with the laser sensor; the filtering module is used to filter the initial point cloud data to obtain intermediate point cloud data; the segmentation module is used to segment the intermediate point cloud data using the RANSAC algorithm to obtain the point cloud data.

[0154] In this solution, the original initial point cloud data can be obtained, and irrelevant information can be eliminated through filtering and segmentation processing to ensure better data quality, so that bolt position identification can be performed more efficiently and accurately in the future.

[0155] During the specific implementation process, the segmentation module includes an acquisition submodule and a clustering submodule. The acquisition submodule is used to obtain the Euclidean distance between the first point cloud and the second point cloud, wherein the first point cloud is any point cloud in the intermediate point cloud data, and the second point cloud is any point cloud in the intermediate point cloud data except the first point cloud; the clustering submodule is used to divide the first point cloud and the second point cloud into a cluster cluster when the Euclidean distance is less than or equal to a preset distance threshold and the total number of point clouds is within a preset range, wherein the total number of point clouds is the sum of the number of the first point cloud and the number of the second point cloud.

[0156] In this scheme, the bolt point clouds in the intermediate point cloud data can be effectively distinguished to form independent clusters to obtain point cloud data with better quality.

[0157] During the specific implementation process, the second acquisition unit includes a second acquisition module, a conversion module, a third acquisition module and an acquisition module. The second acquisition module is used to acquire a CAD model file, wherein the above-mentioned CAD model file is a pre-stored drawing file of the structure of the above-mentioned bolt; the conversion module is used to convert the above-mentioned CAD model file into an STL format to obtain an STL format file; the third acquisition module is used to acquire the three-dimensional geometric information of the above-mentioned STL format file, and perform triangulated meshing processing to obtain a network model file; the acquisition module is used to adopt a uniform sampling algorithm to collect a target number of points from the above-mentioned network model file to obtain the above-mentioned template point cloud data.

[0158] In this solution, by converting the CAD model into STL format, performing triangulated meshing and uniform sampling, the generated template point cloud data can more accurately reflect the true geometric shape of the bolt. The uniform sampling algorithm ensures the uniform distribution of the template point cloud data. Even if some areas of the bolt are obscured or damaged, it can still be matched based on the point cloud features of other complete areas, thereby improving the robustness of recognition.

[0159] In some embodiments, the identification unit includes a calculation module, an extraction module and a fourth acquisition module. The calculation module is used to calculate the similarity between the above-mentioned point cloud data and the above-mentioned template point cloud data; the extraction module is used to extract the points in the above-mentioned point cloud data that are most similar to the above-mentioned template point cloud data to obtain matching points; the fourth acquisition module is used to obtain the position of the above-mentioned matching points in the above-mentioned point cloud data to obtain the position of the above-mentioned bolt.

[0160] In this scheme, the point cloud data obtained in the scene will be compared with the template point cloud data of the bolt, and the similarity between the two will be calculated. The points with the highest similarity to the points in the template point cloud will be found from the scene point cloud. These points will be identified as matching points. The actual position of the bolt on the electric wheel can be determined through the position information of the matching points in the scene point cloud. This scheme can identify the position of the bolt on the electric wheel of the mining truck and achieve high-precision positioning.

[0161] In some embodiments, the above-mentioned device also includes a calibration unit and a scanning unit. The calibration unit is used to calibrate the above-mentioned laser sensor before obtaining the point cloud data of the current mining card; the scanning unit is used to use the calibrated above-mentioned laser sensor to scan the above-mentioned mining card.

[0162] In this solution, calibration can eliminate the hardware error of the sensor and ensure that the acquired point cloud data accurately corresponds to the real-world position. The calibration process ensures the accuracy of the data acquired by the sensor.

[0163] In some embodiments, the above-mentioned device also includes a conversion unit and a control unit. The conversion unit is used to convert the position of the above-mentioned bolt into a target coordinate system after globally matching the above-mentioned point cloud data and the above-mentioned template point cloud data to obtain the position of the above-mentioned bolt, wherein the above-mentioned target coordinate system is a robot coordinate system; the control unit is used to control the robot to disassemble and / or tighten the above-mentioned bolt in the above-mentioned target coordinate system.

[0164] In this solution, in the machine vision system, the point cloud data acquired by the sensor (such as a line laser vision sensor) is in the sensor's own coordinate system. However, the robot control system uses an independent coordinate system, usually called the robot coordinate system. In order for the robot to understand and locate the bolt, the position information of the bolt in the sensor coordinate system must be converted to the robot coordinate system. Once the position of the bolt is converted to the robot coordinate system, the robot control system can accurately control the robot's arm or tool based on these coordinate information to achieve automated removal or tightening of the bolt.

[0165] The bolt position recognition device includes a processor and a memory. The first acquisition unit, the second acquisition unit and the recognition unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0166] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of low efficiency in manually identifying the position of the bolts of the mining truck in the prior art is solved by adjusting the kernel parameters.

[0167] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0168] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the bolt position identification method.

[0169] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the method for identifying the bolt position is executed when the program is run.

[0170] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and executable on the processor, and the processor implements at least the steps of the bolt position identification method when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0171] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the bolt position identification method in each embodiment of the present application are implemented.

[0172] The present application also provides a bolt identification system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above-mentioned bolt position identification methods.

[0173] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0183] 1) The bolt position identification method of the present application identifies the position of the bolt by performing global template matching between the extracted scene point cloud data and the generated template point cloud data. Manual identification of the bolt position is no longer required, which shortens the identification time, reduces the operation risk of workers, achieves high-precision detection of bolts, and improves the efficiency of bolt position identification.

[0184] 2) The bolt position identification device of the present application identifies the position of the bolt by performing global template matching between the extracted point cloud data of the scene and the generated template point cloud data. Manual identification of the bolt position is no longer required, which shortens the identification time, reduces the operation risk of workers, achieves high-precision detection of bolts, and improves the efficiency of bolt position identification.

[0185] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying a bolt position, characterized in that: include: Acquire the point cloud data of the current mining truck, wherein the point cloud data is obtained based on laser sensor scanning; Acquire template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template; The point cloud data and the template point cloud data are globally matched to obtain the position of the bolt.

2. The method according to claim 1, characterized in that: Get the point cloud data of the current mining card, including: Acquire initial point cloud data, wherein the initial point cloud data is initial data obtained by scanning with the laser sensor; Performing filtering processing on the initial point cloud data to obtain intermediate point cloud data; The intermediate point cloud data is segmented using a RANSAC algorithm to obtain the point cloud data.

3. The method according to claim 2, characterized in that The intermediate point cloud data is segmented using the RANSAC algorithm to obtain the point cloud data, including: Obtaining a Euclidean distance between a first point cloud and a second point cloud, wherein the first point cloud is any point cloud in the intermediate point cloud data, and the second point cloud is any point cloud in the intermediate point cloud data except the first point cloud; When the Euclidean distance is less than or equal to a preset distance threshold and the total number of point clouds is within a preset range, the first point cloud and the second point cloud are divided into a cluster, wherein the total number of point clouds is the sum of the number of the first point cloud and the number of the second point cloud.

4. The method according to claim 1, characterized in that Get template point cloud data, including: Acquire a CAD model file, wherein the CAD model file is a pre-stored drawing file of the structure of the bolt; Convert the CAD model file into STL format to obtain an STL format file; Obtaining the three-dimensional geometric information of the STL format file, and performing triangulated mesh processing to obtain a network model file; A uniform sampling algorithm is used to collect a target number of points from the network model file to obtain the template point cloud data.

5. The method according to claim 1, characterized in that Globally matching the point cloud data with the template point cloud data to obtain the position of the bolt includes: Calculating the similarity between the point cloud data and the template point cloud data; Extracting the point in the point cloud data that has the highest similarity to the template point cloud data to obtain a matching point; The position of the matching point in the point cloud data is obtained to obtain the position of the bolt.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the point cloud data of the current mining truck, the method further includes: Calibrate the laser sensor; The calibrated laser sensor is used to scan the mining truck.

7. The method according to any one of claims 1 to 5, characterized in that After globally matching the point cloud data with the template point cloud data to obtain the position of the bolt, the method further includes: Converting the position of the bolt into a target coordinate system, wherein the target coordinate system is a robot coordinate system; In the target coordinate system, the robot is controlled to perform disassembly and / or tightening operations on the bolts.

8. A bolt position identification device, characterized in that: include: A first acquisition unit is used to acquire point cloud data of the current mining truck, wherein the point cloud data is obtained based on scanning by a laser sensor; A second acquisition unit is used to acquire template point cloud data, wherein the template point cloud data is data of a pre-configured bolt template; The recognition unit is used to globally match the point cloud data with the template point cloud data to obtain the position of the bolt.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the bolt position identification method described in any one of claims 1 to 7 are implemented.

10. A bolt identification system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the bolt position identification method described in any one of claims 1 to 7.

Images