A method and system for robot identification and grasping wire harness based on point cloud clustering

Abstract

The application discloses a kind of robot identification and grab harness method and system based on point cloud clustering, it is related to robot control technical field.The method includes the following steps: obtaining the harness image of harness to be grabbed, pre-processes harness image and obtains point cloud data;Point cloud data is segmented using clustering aggregation algorithm, and a plurality of segmented capture areas are obtained;Calculate the capture point of each capture area, determine the end point of harness based on Euclidean distance, and sort the capture point to determine the harness path;Dual-arm robot selects capture point for each arm according to harness path, and performs capture task according to capture point.The application considers the collaborative ability of dual-arm robot harness identification and capture, calculates and selects the actual capture point of automobile harness based on clustering algorithm, and realizes accurate capture of automobile flexible harness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot recognition and grasping wire harness method and system based on point cloud clustering. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the advancement of the intelligent transformation of the automobile manufacturing industry, the importance of wire harness layout on the automobile assembly line is increasingly prominent, and the automation degree has become a key measure of the efficiency and quality of vehicle assembly. However, the current wire harness assembly process in the industry still relies heavily on manual operation, which has brought about many difficult problems. On the one hand, manual assembly of wire harness has a high labor intensity, and workers need to maintain concentration for a long time and repeat fine operations, which is extremely tiring and leads to low work efficiency. On the other hand, due to the subjectivity and individual differences of manual work, it is difficult to ensure the consistency of operation, which makes the assembly quality uneven and poses a hidden danger to the performance and safety of the subsequent vehicle.

[0004] In the prior art, although automatic solutions for wire harness grasping are constantly emerging, there are still two insurmountable technical bottlenecks. First, wire harness is a slender and flexible object, and its pose in three-dimensional space exhibits strong non-rigid characteristics. This means that traditional machine vision algorithms often struggle to effectively identify its features when faced with wire harness. Especially in the absence of obvious features for grasping, the difficulty of identification increases exponentially, making it almost impossible to accurately identify the specific position and attitude of the wire harness, thereby severely restricting the precision and reliability of automated grasping. Second, when multiple branches of the wire harness are entangled in space, the limitations of the single-arm robot end effector are exposed. Due to the lack of collaborative operation capability, the single-arm robot is not up to the task when faced with multi-target separation tasks. It is difficult to simultaneously perform precise and effective separation and grasping operations on multiple entangled wire harness branches, which not only reduces the grasping efficiency but also can cause damage to the wire harness, further affecting the smoothness and quality control of the entire assembly process. The existence of these technical bottlenecks hinders the automation process of wire harness assembly and becomes a difficult problem to be overcome in the intelligent transformation of the automobile manufacturing industry. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application aims to provide a robot recognition and grasping wire harness method and system based on point cloud clustering, which considers the collaborative ability of dual-arm robot wire harness recognition and grasping, calculates and selects the actual grasping points of the automobile wire harness based on clustering algorithms, and realizes accurate grasping of automobile flexible wire harness.

[0006] In order to achieve the above object, the present application is realized by the following technical scheme:

[0007] The first aspect of the present application provides a method for identifying and grabbing wire harness based on point cloud clustering, comprising the following steps:

[0008] Obtaining a wire harness image of a wire harness to be grabbed, and pre-processing the wire harness image to obtain point cloud data;

[0009] Segmenting the point cloud data by a clustering aggregation algorithm to obtain a plurality of segmented grabbing regions;

[0010] Calculating a grabbing point for each grabbing region, determining a wire harness endpoint based on Euclidean distance, and sorting the grabbing points to determine a wire harness path;

[0011] Using a dual-arm robot to select a grabbing point for each arm according to the wire harness path, and performing a grabbing task according to the grabbing point.

[0012] Further, the wire harness image is captured by a 3D camera, and the point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

[0013] Further, the specific steps of pre-processing the wire harness image are:

[0014] Segmenting the wire harness and the background in the wire harness image by a binary threshold method to generate a binary mask that only retains the wire harness region;

[0015] Filtering out background pixels based on the binary mask, and extracting and constructing a two-dimensional point cloud data set containing only wire harness region pixel coordinates.

[0016] Further, the specific steps of segmenting the point cloud data by the clustering aggregation algorithm are:

[0017] Initializing the point cloud data;

[0018] Merging the two-dimensional point cloud data set into a plurality of sub-clusters by a loop merging, each sub-cluster corresponding to a grabbing region.

[0019] Further, the centroid of each grabbing region is selected as the grabbing point.

[0020] Further, before performing the grabbing task according to the wire harness path, the dual-arm robot first initializes the state and resets to a preset safe pose, and then uses a Cartesian space position algorithm to control the movement of the mechanical arm to the target wire harness directly above according to the selected grabbing point through the collaborative planning of the dual arms.

[0021] Further, when performing the grabbing task according to the grabbing point, if the wire harness is entangled, the grabbing point is reselected.

[0022] The second aspect of the present application provides a robot identification and wire harness grabbing system based on point cloud clustering, comprising:

[0023] A visual perception module configured to acquire a wire harness image of a wire harness to be grabbed, and to preprocess the wire harness image to obtain point cloud data;

[0024] A region segmentation module configured to perform region segmentation on the point cloud data using a clustering aggregation algorithm to obtain a plurality of segmented grabbing regions;

[0025] A grabbing point generation module configured to calculate a grabbing point for each grabbing region, determine wire harness endpoints based on Euclidean distance, and sort the grabbing points to determine a wire harness path;

[0026] A dual-arm robot module configured to use a dual-arm robot to select a grabbing point for each arm according to the wire harness path, and perform a grabbing task according to the grabbing points.

[0027] The third aspect of the present application provides a computer readable storage medium storing a computer program, the computer program being adapted to be loaded and executed by a processor to perform the steps of the robot identification and wire harness grabbing method based on point cloud clustering as described in the first aspect of the present application.

[0028] The fourth aspect of the present application provides a computer device, comprising:

[0029] A processor adapted to execute a computer program;

[0030] A computer readable storage medium storing a computer program, the computer program being executed by the processor to implement the robot identification and wire harness grabbing method based on point cloud clustering as described in the first aspect of the present application.

[0031] The above one or more technical solutions have the following beneficial effects:

[0032] The application discloses a robot recognition and grabbing wire harness method and system based on point cloud clustering.

[0033] The application divides the wire harness area by clustering the two-dimensional point cloud data set into a plurality of sub-clusters, each of which corresponds to a grabbing area.

[0034] The application adopts the dual-arm robot to operate the flexible object such as the wire harness, adopts the Cartesian space position control to execute the grabbing task, divides the wire harness into different areas according to the clustering algorithm, then calculates the centroid of the grabbing area, obtains the three-dimensional information of the grabbing point, and selects one grabbing point in the three-dimensional information of the grabbing point for grabbing by one arm of the dual-arm robot, and selects another grabbing point for grabbing by the other arm of the dual-arm robot.

[0035] The application improves the dynamic recognition efficiency of the robot for the wire harness by using the clustering robot learning algorithm, and can adapt to the grabbing operation of the wire harness in different scenes by using the activity range of the dual-arm robot and combining the dynamic grabbing point.

[0036] Advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be learned by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0038] Figure 1 Flow chart of the method for robot to recognize and grab wire harness based on point cloud clustering in the first embodiment of the present application;

[0039] Figure 2 Flow chart of the clustering and aggregation method in the first embodiment of the present application;

[0040] Figure 3 Flow chart of the action execution of the dual-arm robot in the first embodiment of the present application;

[0041] Figure 4 System framework diagram of the robot to recognize and grab wire harness based on point cloud clustering in the second embodiment of the present application. DETAILED DESCRIPTION

[0042] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0043] It should be noted that the terms used herein are only intended to describe specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof;

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] Embodiment I:

[0046] The first embodiment of the present application provides a method for robot to recognize and grab wire harness based on point cloud clustering, as shown in the following. Figure 1As shown, first, the image is photographed to obtain the wire harness point cloud data, then according to the clustering algorithm, the number of sub-clusters is segmented (the grabbing area) according to the distance between clusters, combined with the given grabbing area, the three-dimensional information of the grabbing points in the area is calculated, and these inputs are given to the dual-arm robot. The dual-arm robot performs grabbing operation on the wire harness according to the order of the grabbing points and in accordance with the principle of cooperative motion.

[0047] Specifically, the following steps are included:

[0048] Step 1: Obtain the wire harness image of the wire harness to be grabbed, and pre-process the wire harness image to obtain point cloud data.

[0049] Step 1.1: Obtain the wire harness image of the wire harness to be grabbed.

[0050] In this embodiment, the wire harness image is photographed by a 3D camera, and the wire harness image contains 2D point cloud data and visual information of the RGB image. The point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

[0051] Step 1.2: Preprocess the wire harness image based on the computer vision algorithm of OpenCV.

[0052] Step 1.2.1: Use binary threshold method to segment the wire harness and background in the wire harness image, and generate a binary mask that only retains the wire harness area.

[0053] Step 1.2.2: Based on the binary mask, filter out the background pixels, extract and construct a two-dimensional point cloud dataset containing only the pixel (X, Y) coordinates of the wire harness area.

[0054] Step 2: Use clustering aggregation algorithm to segment the point cloud data to obtain multiple grabbing areas after segmentation.

[0055] Step 2.1: Perform initialization operation on the point cloud data.

[0056] In a specific embodiment, the number of clusters is defined as N, and the number of clusters represents the number of final grabbing areas, i.e. the number of grabbing points on the wire harness. The distance between each of the n point cloud data samples in the two-dimensional point cloud dataset is defined. The method for calculating the merging class is defined, and the ward variance method is used in this embodiment.

[0057] Construct n classes, each sample as a class. Calculate the Euclidean distance between each of the n samples , denoted as .

[0058] Step 2.2: Merge the two-dimensional point cloud data into several sub-clusters through loop merging, and each sub-cluster corresponds to a grabbing area.

[0059] In a specific embodiment, as shown in Figure 2 The two classes with the smallest distance between classes in the point cloud data are merged based on the ward variance method to construct a new class. It is determined whether the number of classes reaches the number of clusters. If it is reached, the aggregation result is output. If it is not reached, the two classes with the smallest distance between classes are merged based on the ward variance method to construct a new class, and the loop is determined. In the merging process, the measurement value of the k-nearest neighbor method is k. The Euclidean distance between the to-be-merged point cloud data sample and other samples is calculated, and the to-be-merged point cloud data sample is merged into the class with a high proportion of k distance values.

[0060] The iteration formula of the ward variance method is as follows:

[0061] .

[0062] Wherein is the distance between the class and the new class , represents the number of samples of , , the new class is formed by merging the classes , . , , are all initial classes.

[0063] Step 3: Calculate the grabbing point of each grabbing area, determine the wire bundle end point based on the Euclidean distance, and sort the grabbing point to determine the wire bundle path.

[0064] In a specific embodiment, the grabbing point is calculated according to the area of the N sub-clusters, and the three-dimensional information of the grabbing point is output. In this embodiment, the centroid of each grabbing area is selected as the grabbing point.

[0065] Specifically, the clustering aggregation algorithm outputs an aggregated point cloud area for the sample set, the number of areas is clusters, and each aggregated point cloud area corresponds to a grabbing area. The centroid of each grabbing area is calculated, which is the grabbing point of the area. The centroid calculation formula is as follows:

[0066] .

[0067] Wherein, is the centroid of the kth cluster, is the total number of points in the cluster, is the number of points in the cluster, is the horizontal coordinate of the mth point, is the vertical coordinate of the mth point. is the vertical coordinate of the mth point.​

[0068] The embodiment sorts the grabbing points. Specifically, all the grabbing points are input, the Euclidean distance of each two grabbing points is calculated, the two points with the maximum distance are selected as the two endpoints of the bundle, denoted as A and B, and the sorting is performed from the endpoint A to the endpoint B, so that the grabbing points, i.e. the paths of the corresponding bundles, are sequentially arranged.

[0069] Step 4: The dual-arm robot selects the grabbing points for each arm according to the bundle path, and performs the grabbing task according to the grabbing points.

[0070] Before the dual-arm robot performs the grabbing task according to the bundle path, the state is initialized, and the robot is reset to the preset safe pose. Subsequently, the Cartesian space position algorithm is adopted to control the movement of the robot to the target bundle above according to the selected grabbing points through the collaborative planning of the dual arms. Specifically, the Cartesian space position algorithm is as follows: according to the coordinates of the grabbing points, a smooth motion path of the end effector from the starting point to the target point is planned in the Cartesian space, and the origin (TCP) of the end tool coordinate system moves along the straight line between the two points with a constant pose.

[0071] When performing the grabbing task according to the grabbing points, one arm selects one grabbing point for grabbing, and the other arm selects another grabbing point for grabbing. If the bundle is entangled, the grabbing points are reselected to separate the entanglement.

[0072] It should be particularly noted that the bundle grabbed in the embodiment includes one main branch and several sub-branches. The entanglement refers to the covering phenomenon of the sub-branches of the bundle, i.e. one sub-branch of the bundle is above another sub-branch, which can be solved by reselecting the grabbing points.

[0073] According to the placement of the bundle, the following entanglement phenomena are prone to occur:

[0074] 1. When the bundle is not placed flat in the initial stage, only one robot arm grabs, and the other robot arm gives up the grabbing of the other endpoint, and performs the grabbing in the area before and after the entanglement position. The grabbing area is determined according to the current position of the robot, and the position close to the robot is preferentially selected. According to the path generated by the previous sorting, the bundle is straightened, and then the grabbing is performed from the other endpoint. During this period, if the bundle is too long and exceeds the activity range of the robot, the robot arm that starts the grabbing moves according to the grabbing point close to the endpoint, so that the robot arm remains in the activity range.

[0075] 2. When the bundle is placed flat in the initial stage, the entanglement is the entanglement of the sub-branches of the bundle, rather than the entanglement of the main branch of the bundle. In this case, only the endpoint of the sub-branch needs to be grabbed to make it separate from the main branch, so that the subsequent operation can be performed.

[0076] Embodiment Two:

[0077] Embodiment two of the present application provides a robot identification and grabbing wire harness system based on point cloud clustering, as shown in the drawings, comprising: Figure 4

[0078] a visual perception module configured to acquire a wire harness image of a wire harness to be grabbed, and to preprocess the wire harness image to obtain point cloud data;

[0079] a region segmentation module configured to use a clustering aggregation algorithm to perform region segmentation on the point cloud data to obtain a plurality of segmented grabbing regions;

[0080] a grabbing point generation module configured to calculate a grabbing point for each grabbing region, determine wire harness end points based on Euclidean distance, and sort the grabbing points to determine a wire harness path;

[0081] a dual-arm robot module configured to use a dual-arm robot to select a grabbing point for each arm according to the wire harness path, and perform a grabbing task according to the grabbing points.

[0082] Embodiment three:

[0083] Embodiment three of the present application provides a computer readable storage medium storing a computer program, the computer program being adapted to be loaded and executed by a processor to perform the steps of the robot identification and grabbing wire harness method based on point cloud clustering as described in embodiment one of the present application.

[0084] Embodiment four:

[0085] Embodiment four of the present application provides a computer device, comprising:

[0086] a processor adapted to execute a computer program;

[0087] a computer readable storage medium storing a computer program, the computer program being executed by the processor to implement the steps of the robot identification and grabbing wire harness method based on point cloud clustering as described in embodiment one of the present application.

[0088] The steps and methods of embodiments two, three and four correspond to the steps and methods of embodiment one, and the specific embodiments can refer to the relevant description of embodiment one.

[0089] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed in the present application can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. ​

[0090] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable device. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data processing device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD) or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0091] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for robot identification and grasping wire harness based on point cloud clustering, characterized in that, The method comprises the following steps: An image of a wire harness to be grabbed is acquired, and point cloud data is acquired by preprocessing the image of the wire harness; A region segmentation is performed on the point cloud data by using a clustering aggregation algorithm to obtain a plurality of segmented grabbing regions; The specific steps of performing the region segmentation on the point cloud data by using the clustering aggregation algorithm are as follows: An initialization operation is performed on the point cloud data, specifically as follows: The number of clustering levels clusters is defined as N, and the number of clusters represents the number of final grabbing regions; the distance between each two of n point cloud data samples in a two-dimensional point cloud data set is defined; a method for calculating a merging class is defined, and the ward variance method is adopted; The two-dimensional point cloud data set is merged into a plurality of sub-clusters through a loop merging, and each sub-cluster corresponds to a grabbing region, specifically as follows: Two classes with the smallest inter-class distance in the point cloud data are merged based on the ward variance method to construct a new class, and it is judged whether the number of classes reaches the number of clusters; if yes, the aggregation result is output; if no, two classes with the smallest inter-class distance are merged based on the ward variance method to construct a new class, and the loop judgment is performed; in the merging process, the measurement value of the k-nearest neighbor method is k, the Euclidean distance between the point cloud data sample to be merged and other samples is calculated, and the point cloud data sample to be merged is merged into the class with a high proportion in the k distance values; The iteration formula of the ward variance method is as follows: wherein is a class distance to a new class , represent number of samples, , new class is merged from classes , , , , are initial classes A grabbing point of each grabbing region is calculated, the end point of the wire harness is determined based on the Euclidean distance, and the grabbing point is sorted to determine a wire harness path; A double-arm robot selects a grabbing point for each arm according to the wire harness path, and performs a grabbing task according to the grabbing point.

2. The method of claim 1, wherein, A wire harness image is captured by a 3D camera, and point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

3. The method of claim 1, wherein, The specific steps of preprocessing the wire harness image are as follows: The wire harness and the background in the wire harness image are segmented by using a binary threshold method to generate a binary mask that only retains the wire harness region; Based on the binary mask, background pixels are filtered out, and a two-dimensional point cloud data set containing only pixel coordinates of the wire harness region is extracted and constructed.

4. The method of claim 1, wherein, The centroid of each grabbing region is selected as a grabbing point.

5. The method of claim 1, wherein, Before performing a grabbing task according to the wire harness path, the double-arm robot is first initialized in a state, reset to a preset safe pose, and then controlled by a collaborative planning algorithm in a Cartesian space position to move to a target wire harness directly above the target wire harness to perform grabbing according to the selected grabbing point.

6. The method of claim 1, wherein, When performing a grabbing task according to the grabbing point, if the wire harness is entangled, the grabbing point is reselected.

7. A system for robot identification and grasping of a wire harness based on point cloud clustering, the system comprising: It comprises: A visual perception module configured to acquire an image of a wire harness to be grabbed, and to acquire point cloud data by preprocessing the image of the wire harness; A region segmentation module configured to perform a region segmentation on the point cloud data by using a clustering aggregation algorithm to obtain a plurality of segmented grabbing regions; The specific steps of performing the region segmentation on the point cloud data by using the clustering aggregation algorithm are as follows: An initialization operation is performed on the point cloud data, specifically as follows: Define the number of clusters as N, which represents the number of final grasping regions; define the distance between each two point cloud data samples in the two-dimensional point cloud data set; define the method for calculating the merging class, using the ward variance method; The two-dimensional point cloud data set is merged into several sub-clusters through loop merging, and each sub-cluster corresponds to a grasping region, specifically: Based on the ward variance method, the two classes with the smallest inter-class distance in the point cloud data are merged to construct a new class, and it is judged whether the number of classes reaches the number of clusters. If it does, the aggregation result is output; if it does not, the two classes with the smallest inter-class distance are merged based on the ward variance method to construct a new class, and the loop is judged. In the merging process, the measurement value of the k-nearest neighbor method is k, the Euclidean distance between the to-be-merged point cloud data sample and other samples is calculated, and the to-be-merged point cloud data sample is merged into the class with a high proportion of k distance values. The iteration formula of the ward variance method is as follows: wherein is a class distance to a new class , represent number of samples, , new class is formed by merging classes , , , , are initial classes The grasping point generation module is configured to calculate the grasping point of each grasping region, determine the bundle end point based on the Euclidean distance, and sort the grasping points to determine the bundle path; The dual-arm robot module is configured to use the dual-arm robot to select the grasping point for each arm according to the bundle path, and perform the grasping task according to the grasping point.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by the processor to implement the point cloud clustering based robot identification grasping bundle method of any one of claims 1-6.

9. A computer device, comprising: It includes: A processor suitable for executing a computer program; A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor, and the point cloud clustering based robot identification grasping bundle method of any one of claims 1-6 is realized.

Images