Method and system for recognizing and grabbing wire harness by robot based on point cloud clustering

Abstract

The invention discloses a method and a system for recognizing and grabbing a wire harness by a robot based on point cloud clustering, and relates to the technical field of robot control. The method comprises the following steps: acquiring a wire harness image of a to-be-captured wire harness, and preprocessing the wire harness image to acquire point cloud data; performing region segmentation on the point cloud data by using a clustering aggregation algorithm to obtain a plurality of segmented capture regions; the grabbing points of each grabbing area are calculated, wire harness end points are determined based on the Euclidean distance, the grabbing points are ranked, and a wire harness path is determined; and the double-arm robot is used for selecting a grabbing point for each arm according to the wire harness path, and a grabbing task is carried out according to the grabbing points. According to the method, the cooperative capability of wire harness recognition and grabbing of the two-arm robot is considered, the actual grabbing points of the automobile wire harness are calculated and selected based on the clustering algorithm, and accurate grabbing of the automobile flexible wire harness is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot recognition and grasping wire harness method and system based on point cloud clustering. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the advancement of the intelligent transformation of the automobile manufacturing industry, the importance of wire harness layout on the automobile assembly line is increasingly prominent, and the automation degree has become a key measure of the efficiency and quality of vehicle assembly. However, the current wire harness assembly process in the industry still relies heavily on manual operation, which has brought about many difficult problems. On the one hand, manual assembly of wire harness has a high labor intensity, and workers need to maintain concentration for a long time and repeat fine operations, which is extremely tiring and leads to low work efficiency. On the other hand, due to the subjectivity and individual differences of manual work, it is difficult to ensure the consistency of operation, which makes the assembly quality uneven and poses a hidden danger to the performance and safety of the subsequent vehicle.

[0004] In the prior art, although automatic solutions for wire harness grasping are constantly emerging, there are still two insurmountable technical bottlenecks. First, wire harness is a slender and flexible object, and its pose in three-dimensional space exhibits strong non-rigid characteristics. This means that traditional machine vision algorithms often struggle to effectively identify its features when faced with wire harness. Especially in the absence of obvious features for grasping, the difficulty of identification increases exponentially, making it almost impossible to accurately identify the specific position and attitude of the wire harness, thereby severely restricting the precision and reliability of automated grasping. Second, when multiple branches of the wire harness are entangled in space, the limitations of the single-arm robot end effector are exposed. Due to the lack of collaborative operation capability, the single-arm robot is not up to the task when faced with multi-target separation tasks. It is difficult to simultaneously perform precise and effective separation and grasping operations on multiple entangled wire harness branches, which not only reduces the grasping efficiency but also can cause damage to the wire harness, further affecting the smoothness and quality control of the entire assembly process. The existence of these technical bottlenecks hinders the automation process of wire harness assembly and becomes a difficult problem to be overcome in the intelligent transformation of the automobile manufacturing industry. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application aims to provide a robot recognition and grasping wire harness method and system based on point cloud clustering, which considers the collaborative ability of dual-arm robot wire harness recognition and grasping, calculates and selects the actual grasping points of the automobile wire harness based on clustering algorithms, and realizes accurate grasping of automobile flexible wire harness.

[0006] In order to achieve the above object, the present application is realized by the following technical scheme: The first aspect of the present application provides a method for robot recognition and grabbing wire harness based on point cloud clustering, comprising the following steps: Obtaining a wire harness image of a wire harness to be grabbed, and pre-processing the wire harness image to obtain point cloud data; Using a clustering aggregation algorithm to perform regional segmentation on the point cloud data to obtain a plurality of segmented grabbing regions; Calculating a grabbing point for each grabbing region, determining a wire harness endpoint based on the Euclidean distance, and sorting the grabbing points to determine a wire harness path; Using a dual-arm robot to select a grabbing point for each arm according to the wire harness path, and performing a grabbing task according to the grabbing point.

[0007] Further, the wire harness image is captured by a 3D camera, and the point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

[0008] Further, the specific steps of pre-processing the wire harness image are: Using a binary threshold method to segment the wire harness from the background in the wire harness image to generate a binary mask that only retains the wire harness region; Filtering out background pixels based on the binary mask, and extracting and constructing a two-dimensional point cloud data set containing only the pixel coordinates of the wire harness region.

[0009] Further, the specific steps of using a clustering aggregation algorithm to perform regional segmentation on the point cloud data are: Performing an initialization operation on the point cloud data; Merging the two-dimensional point cloud data set into a plurality of sub-clusters through a loop, each sub-cluster corresponding to a grabbing region.

[0010] Further, the centroid of each grabbing region is selected as the grabbing point.

[0011] Further, before performing a grabbing task according to the wire harness path, the dual-arm robot first performs state initialization and resets to a preset safe pose, and then uses a Cartesian space position algorithm to move the mechanical arm to the target wire harness directly above according to the selected grabbing point through collaborative planning of the dual arms to perform grabbing.

[0012] Further, when performing a grabbing task according to the grabbing point, if the wire harness is entangled, the grabbing point is reselected.

[0013] The second aspect of the present application provides a system for robot recognition and grabbing wire harness based on point cloud clustering, comprising: A visual perception module configured to obtain a wire harness image of a wire harness to be grabbed, and pre-process the wire harness image to obtain point cloud data; The region segmentation module is configured to segment the point cloud data by using a clustering aggregation algorithm to obtain a plurality of segmented gripping regions; The gripping point generation module is configured to calculate a gripping point of each gripping region, determine a wire harness end point based on an Euclidean distance, and sort the gripping points to determine a wire harness path. The dual-arm robot module is configured to select a gripping point for each arm according to the wire harness path by using a dual-arm robot, and perform a gripping task according to the gripping point.

[0014] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor to implement the steps in the method for identifying and gripping a wire harness based on point cloud clustering of the first aspect of the present application.

[0015] The fourth aspect of the present application provides a computer device, which comprises: A processor suitable for executing a computer program; A computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to implement the method for identifying and gripping a wire harness based on point cloud clustering of the first aspect of the present application.

[0016] The above one or more technical solutions have the following beneficial effects: The present application discloses a method and system for identifying and gripping a wire harness based on point cloud clustering. Firstly, real-time RGB and depth images are captured, and only the (X, Y) coordinates of the wire harness region pixels are reserved based on OpenCV computer vision preprocessing to construct a two-dimensional point cloud dataset. Then, the two-dimensional point cloud dataset of the wire harness is aggregated by using a condensation clustering machine learning algorithm to segment the wire harness point cloud and output N sub-clusters corresponding to N gripping regions. Then, the centroid of the gripping region is calculated based on the obtained gripping region to output three-dimensional information of the gripping point. Finally, the dual-arm robot performs a gripping action in Cartesian space position control according to the three-dimensional information of the gripping point to complete the task operation. The wire harness is segmented into different regions by using the clustering algorithm, and the gripping point is determined, which avoids the problem that the wire harness is difficult to identify as a slender flexible object, and improves the identification ability of the robot for the wire harness and other slender flexible objects.

[0017] The application realizes the division of the wire harness area by merging two-dimensional point cloud data into multiple sub-clusters, each of which corresponds to a grabbing area. This method can adjust the number of sub-clusters according to the actual needs of grabbing, and then determine the grabbing points on the wire harness, so that the dual-arm robot can grab according to these points. This method not only overcomes the defect that the traditional visual technology cannot accurately identify the wire harness, but also solves the problem that the collaborative grabbing points cannot be determined when the dual-arm robot grabs the wire harness, laying a foundation for subsequent wire harness wiring and automobile assembly links.

[0018] The application adopts a dual-arm robot to operate flexible objects such as wire harnesses, performs a grabbing task by adopting Cartesian space position control, divides the wire harness into different areas according to a clustering algorithm, then calculates the centroid of the grabbing area to obtain three-dimensional information of the grabbing points, one arm of the dual-arm robot selects one grabbing point in the three-dimensional information of the grabbing points to perform grabbing, and the other arm of the dual-arm robot selects another grabbing point to perform grabbing. When the wire harness is entangled during grabbing, the dual-arm robot can select another grabbing point to perform grabbing again to separate the entanglement, thereby avoiding a large amount of interaction between the robot and the wire harness and improving the ability of the robot to operate flexible objects such as wire harnesses.

[0019] The application utilizes a clustering robot learning algorithm to improve the efficiency of dynamic identification of wire harnesses by the robot, and utilizes the activity range of the dual-arm robot in combination with dynamic grabbing points to adapt to the grabbing operation of the wire harness in different scenarios.

[0020] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be learned through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, without creative labor, other drawings can also be obtained from these drawings.

[0022] Figure 1 The flow chart of the robot identification and grabbing wire harness method based on point cloud clustering in the embodiment one of the application; Figure 2 The flow chart of the clustering aggregation method in the embodiment one of the application; Figure 3 The flow chart of the dual-arm robot action execution in the embodiment one of the application; Figure 4 The system framework diagram of the robot identification and grabbing wire harness based on point cloud clustering in the embodiment two of the application. DETAILED DESCRIPTION

[0023] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0024] It is also important to note that the use of the term "example" herein is used in the sense of serving as an example, instance, or illustration. Thus, any implementation described herein as "example" is not necessarily to be construed as preferred or advantageous over other implementations. Unless otherwise indicated, the use of the singular terms "a", "an" and "the" are intended to refer to the singular as well as to the plural. The use of the term "and / or" is intended to represent either items that are conjunctively present (i.e., items "and" another item) or items that are disjunctively present (i.e., items "or" another item). The use of the term "at least one of' is intended to mean one or more of an item: the items "comprise", "comprises" and "comprising" are inclusive or open-ended and do not exclude additional, unrecited elements or method steps. The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0025] Embodiment one The embodiment one of the present application provides a robot recognition and grabbing wire harness method based on point cloud clustering, as shown in the figure, first, an image is taken to obtain point cloud data of the wire harness, then according to a clustering algorithm, the number of sub-clusters (grabbing area) is segmented according to the rule of distance between clusters, combined with the given grabbing area, the three-dimensional information of the grabbing point in the area is calculated, the input is given to the dual-arm robot, the dual-arm robot performs grabbing operation on the wire harness according to the order of the grabbing point and in compliance with the principle of cooperative movement. Figure 1 Specifically, the following steps are included:

[0026] Step 1: Obtain the wire harness image of the wire harness to be grabbed, and pre-process the wire harness image to obtain point cloud data. Step 1.1: Obtain the wire harness image of the wire harness to be grabbed.

[0027] In this embodiment, the wire harness image is taken by a 3D camera, and the wire harness image contains 2D point cloud data and visual information of the RGB image of the wire harness. The point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

[0028] Step 1.2: Preprocess the wire harness image based on the computer vision algorithm of OpenCV.

[0029] Step 1.2.1: Use binary threshold method to segment the wire harness and background in the wire harness image to generate a binary mask that only retains the wire harness area.

[0030]

[0031] ​Step 1.2.2: Filter out background pixels based on the binary mask, and extract and construct a two-dimensional point cloud dataset containing only the (X, Y) coordinates of the pixels in the harness area.

[0032] Step 2: Use clustering aggregation algorithm to segment the point cloud data into regions and obtain multiple grasping areas after segmentation.

[0033] Step 2.1: Initialize the point cloud data.

[0034] In one specific embodiment, the number of clusters at the clustering level is defined as N. The number of clusters represents the number of final grasping areas, i.e., the number of grasping points on the harness. The distance between each of the n point cloud data samples in the two-dimensional point cloud dataset is defined. A method for calculating the merged categories is defined. In this embodiment, the Ward variance method is used.

[0035] Construct n categories, each sample as a category. Calculate the Euclidean distance between n samples , recorded as .

[0036] Step 2.2: Merge the 2D point cloud dataset into several subclusters through loop merging, where each subcluster corresponds to a grasping area.

[0037] In a specific embodiment, Figure 2 As shown in the figure, the two classes with the smallest inter-class distance in the point cloud data are merged using the Ward variance method to construct a new class. The team then checks whether the number of classes reaches the number of clusters. If so, the aggregated result is output. If not, the Ward variance method is used again to merge the two classes with the smallest inter-class distance to construct a new class, and the process repeats. During the merging process, the k-nearest neighbor method uses a metric of k. The Euclidean distance between the point cloud data sample to be merged and the other samples is calculated, and the point cloud data sample to be merged is merged into the class with the highest proportion of the k distance values.

[0038] The iterative formula of the ward variance method is as follows: .

[0039] in For category With new categories distance, represent The number of samples, , new category By Category , Merged. , , All are initial categories.

[0040] Step 3: Calculate the grasping point of each grasping area, determine the wire harness end point based on the Euclidean distance, and sort the grasping points to determine the wire harness path.

[0041] In a specific embodiment, the grasping point is calculated according to the area of the N sub-clusters, and the three-dimensional information of the grasping point is output. In this embodiment, the centroid of each grasping area is selected as the grasping point.

[0042] Specifically, the clustering aggregation algorithm outputs an aggregated point cloud area for the sample set, and the number of areas is clusters. Each aggregated point cloud area corresponds to a grasping area. The centroid of each grasping area is calculated, which is the grasping point of the area. The centroid calculation formula is as follows: .

[0043] wherein, is the centroid of the kth cluster, is the total number of points in the cluster, is the number of points in the cluster, is the horizontal coordinate of the mth point, is the vertical coordinate of the mth point. In this embodiment, the grasping points are sorted. Specifically, all grasping points are input, the Euclidean distance between each two grasping points is calculated, the two points with the maximum distance are selected as the two end points of the wire harness, denoted as A and B, and then the end point A is taken as the starting point for sorting until the end point B, so that the grasping points are sequentially arranged, i.e. the path of the corresponding wire harness.

[0044]

[0045] Step 4: Use the dual-arm robot to select the grasping point for each arm according to the wire harness path, and perform the grasping task according to the grasping point.

[0046] Before the dual-arm robot performs the grasping task according to the wire harness path, it first initializes the state and resets to a preset safe pose. Then, the Cartesian space position algorithm is used to control the movement of the robot to the target wire harness above according to the selected grasping point through the cooperative planning of the dual arms. Specifically, the Cartesian space position algorithm is as follows: according to the coordinates of the grasping point, a smooth motion path of the end effector from the starting point to the target point is planned in the Cartesian space, and the origin of the end tool coordinate system (TCP) moves along the straight line between the two points with a constant attitude.

[0047] When performing the grasping task according to the grasping point, one arm selects one grasping point for grasping, and the other arm selects another grasping point for grasping. If the wire harness is entangled, the grasping point is reselected to separate the entanglement. ​​It needs to be particularly pointed out that the harness captured in the embodiment includes a main branch and several sub-branches. The winding refers to the covering phenomenon of the sub-branches of the harness, that is, one sub-branch of the harness is above another sub-branch, which can be solved by reselecting the capture point.

[0048] According to the placement of the harness, the following winding phenomena are prone to occur: 1. When the harness is not placed flat in the initial stage, only one mechanical arm captures, and the other mechanical arm gives up the capture of the other end point, captures in the area before and after the winding position, and the capture area is determined according to the current position of the robot, and the position close to the robot is preferentially selected. According to the path generated by the previous sorting, the harness is straightened, and then the other end point is captured. During this period, if the harness is too long and exceeds the activity range of the mechanical arm, the mechanical arm that captures first moves according to the capture point close to the end point, so that the mechanical arm remains in the activity range.

[0049] 2. When the harness is placed flat in the initial stage, the winding is the winding of the sub-branches of the harness, not the winding of the main branch of the harness. In this case, only the end points of the sub-branches need to be captured to make them separate from the main branch, and then subsequent operations can be performed.

[0050] Embodiment two: The embodiment two of the present application provides a robot recognition and capture harness system based on point cloud clustering, as shown in Figure 4 The embodiment two of the present application provides a robot recognition and capture harness system based on point cloud clustering, as shown in The vision perception module is configured to obtain a harness image of a harness to be captured, and to preprocess the harness image to obtain point cloud data; The region segmentation module is configured to perform region segmentation on the point cloud data by using a clustering aggregation algorithm to obtain a plurality of segmented capture regions; The capture point generation module is configured to calculate a capture point of each capture region, determine a harness end point based on Euclidean distance, and sort the capture points to determine a harness path; The dual-arm robot module is configured to select a capture point for each arm according to the harness path by using a dual-arm robot, and perform a capture task according to the capture point.

[0051] Embodiment three: The embodiment three of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor to perform the steps in the robot recognition and capture harness method based on point cloud clustering as described in the embodiment one of the present application.

[0052] Embodiment four: The embodiment four of the present application provides a computer device, which includes: A processor is suitable for executing a computer program; The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps in the robot identification and wire harness grasping method based on point cloud clustering as described in the embodiment one.

[0053] The steps involved in the above embodiments two, three and four correspond to the method embodiment one, and the specific implementation can refer to the related description part of the embodiment one.

[0054] Those skilled in the art can appreciate that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. Computer instructions can be stored in a computer readable storage medium or transmitted by a computer readable storage medium. Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that the computer can access or a data processing device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD) or semiconductor media (for example, solid state disk (SSD)) and the like. The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for robot recognition and grasping of wire harnesses based on point cloud clustering, characterized in that: The following steps are involved: Obtaining a wire harness image of the wire harness to be grabbed, and preprocessing the wire harness image to obtain point cloud data; Use clustering aggregation algorithm to segment point cloud data into regions and obtain multiple grasping areas after segmentation; Calculate the grasping points of each grasping area, determine the endpoints of the harness based on the Euclidean distance, and sort the grasping points to determine the harness path; A dual-arm robot is used to select grasping points for each arm according to the harness path, and the grasping task is performed based on the grasping points.

2. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: The wire harness image is captured by a 3D camera, and the point cloud data of the wire harness in the wire harness image is captured by a binocular camera.

3. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: The specific steps for preprocessing the wire harness image are: The binary threshold method is used to segment the wire harness and the background in the wire harness image, and a binary mask is generated that only retains the wire harness area. Background pixels are filtered out based on the binary mask, and a two-dimensional point cloud dataset containing only the pixel coordinates of the harness area is extracted and constructed.

4. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: The specific steps for using clustering aggregation algorithm to segment point cloud data are as follows: Initialize the point cloud data; The two-dimensional point cloud dataset is merged into several subclusters through cyclic merging, and each subcluster corresponds to a grasping area.

5. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: The centroid of each grasping area is selected as the grasping point.

6. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: Before performing a grasping task according to the harness path, the dual-arm robot first initializes its state and resets to a preset safe position. Then, it uses a Cartesian spatial position algorithm to control the robotic arm to move to the top of the target harness for grasping according to the selected grasping point through collaborative planning of the dual arms.

7. The method for robot recognition and grasping of wire harnesses based on point cloud clustering according to claim 1, characterized in that: When performing a grasping task based on the grasping point, if the harness becomes entangled, reselect the grasping point.

8. A robot recognition and grasping harness system based on point cloud clustering, characterized in that: include: a visual perception module configured to obtain a wire harness image of the wire harness to be grasped and pre-process the wire harness image to obtain point cloud data; The region segmentation module is configured to use a clustering aggregation algorithm to perform region segmentation on the point cloud data to obtain multiple grasping regions after segmentation; a grasping point generation module configured to calculate grasping points of each grasping area, determine the endpoints of the wiring harness based on Euclidean distance, and sort the grasping points to determine the wiring harness path; The dual-arm robot module is configured to use the dual-arm robot to select a grasping point for each arm according to the harness path, and perform grasping tasks based on the grasping point.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the robot recognition and grasping wire harness method based on point cloud clustering according to any one of claims 1 to 7.

10. A computer device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the method for robot recognition and grasping of wire harnesses based on point cloud clustering according to any one of claims 1 to 7 is implemented.

Images