Fastener assembly method, system, device and storage medium

By receiving assembly request instructions and specification data, obtaining the depth map of the assembly object, determining the 3D position data of the center of the assembly hole, controlling the fastener assembly equipment to implant the fastener into the assembly hole, solving the fastener assembly in the papermaking industry The problem of low automation has achieved efficient and good quality fastener assembly.

CN118781126BActive Publication Date: 2024-11-19CHAINT CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411278458.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-19
Estimated Expiration
2044-09-12

AI Technical Summary

Technical Problem

In the papermaking industry, the degree of automation of fastener assembly leads to low assembly quality, high labor costs, high labor intensity, and high defect rate.

Method used

By receiving assembly request instructions and specification data, obtaining the depth map of the assembly object, determining the 3D position data of the center of the assembly hole, controlling the fastener assembly equipment to implant the fastener into the assembly hole, realizing automatic assembly, and detecting the assembly surface Flatness and hole roundness to ensure assembly quality.

Benefits of technology

It improves the degree of automation of fastener assembly, reduces labor intensity, improves production efficiency and assembly quality, and reduces defective rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781126B_ABST
    Figure CN118781126B_ABST
Patent Text Reader

Abstract

The present application provides a fastener assembly method, system, device and storage medium, the method comprising: receiving an assembly request instruction and specification data and / or position data of a fastener; in response to the assembly request instruction, obtaining a first depth map of an assembly object, and grabbing the fastener through a fastener assembly device according to the specification data and / or position data of the fastener; the first depth map includes an assembly hole area of ​​the assembly object; according to the first depth map, determining the 3D pose data of the center of the assembly hole; according to the 3D pose data of the center of the assembly hole, controlling the fastener assembly device to implant the fastener into the assembly hole of the assembly object. The present application can realize that the fastener assembly device automatically grabs the fastener, and automatically identifies the 3D pose data of the center of the assembly hole, and then implants the fastener into the assembly hole, thereby realizing the automated assembly of the fastener, with a high degree of automation, which can greatly improve the assembly quality of the fastener and reduce the defective rate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of machine vision technology, and in particular, relates to a fastener assembly method, system, device and storage medium. Background Art

[0002] In the papermaking industry, the paper rolls produced are large in size and heavy in weight. In order to keep the paper rolls in good shape during storage and transportation so as not to affect subsequent processing, it is generally necessary to insert a wooden or plastic plug into the paper core cavity in the center of the paper roll to prevent the paper rolls from squeezing each other or the paper core of the paper roll from collapsing due to moisture.

[0003] The assembly of the plug and the paper core generally adopts interference fit. In the current actual production application, there are two main types of plug assembly: one is completed manually, which will lead to high labor costs and high labor intensity; the other is completed by a plug assembly device. The current plug assembly device has a low degree of automation and the assembly quality cannot be guaranteed. Summary of the invention

[0004] The embodiments of the present application provide a fastener assembly method, system, device and storage medium, which can solve the current problems of low automation level and low assembly quality of fastener assembly.

[0005] In a first aspect, an embodiment of the present application provides a fastener assembly method, the method comprising:

[0006] receiving an assembly request instruction and specification data and / or location data of a fastener;

[0007] In response to the assembly request instruction, a first depth map of the assembly object is acquired, and according to the specification data and / or position data of the fastener, a fastener is grasped by a fastener assembly device; the first depth map includes an assembly hole area of ​​the assembly object;

[0008] Determining 3D pose data of the center of the assembly hole according to the first depth map;

[0009] According to the 3D position data of the center of the assembly hole, the fastener assembly device is controlled to implant the fastener into the assembly hole of the assembly object.

[0010] In a possible implementation manner of the first aspect, determining 3D pose data of the center of the assembly hole according to the first depth map includes:

[0011] Converting the first depth map into point cloud data to obtain first point cloud data;

[0012] Determine the assembly surface point cloud data within the ROI area according to the first point cloud data;

[0013] Projecting the assembly surface point cloud data within the ROI region into a depth map to obtain a second depth map;

[0014] According to the second depth map, determining the position of the center of the assembly hole by a circle detection algorithm;

[0015] The position of the center of the assembly hole is projected into 3D pose data of the center of the assembly hole.

[0016] In a possible implementation manner of the first aspect, before controlling the fastener assembly device to implant the fastener into the assembly hole according to the 3D pose data of the center of the assembly hole, the method further includes:

[0017] Detecting the flatness of the assembly surface of the assembly object and the roundness of the assembly hole;

[0018] The step of controlling the fastener assembly device to implant the fastener into the assembly hole of the assembly object according to the 3D pose data of the center of the assembly hole comprises:

[0019] If the flatness of the assembly surface of the assembly object meets the first assembly requirement and the roundness of the assembly hole meets the second assembly requirement, the fastener assembly device is controlled to implant the fastener into the assembly hole according to the 3D pose data of the center of the assembly hole.

[0020] In a possible implementation manner of the first aspect, detecting the flatness of the assembly surface of the assembly object and the roundness of the assembly hole includes:

[0021] Taking the center of the assembly hole as the center, extracting point cloud data of a first area around the assembly hole from the first point cloud data; the first area is larger than the assembly hole area and smaller than the ROI area;

[0022] Fitting a point cloud plane within the first region, and determining whether a point cloud within the first region is an inlier or an outlier according to the point cloud plane within the first region and a preset first inlier threshold;

[0023] Determining the flatness of the assembly surface according to the distribution of the point cloud within the first area at the inner points and the outer points;

[0024] Determining whether the assembly surface meets assembly requirements according to the flatness of the assembly surface;

[0025] The roundness of the assembly hole is detected, and whether the assembly hole meets the assembly requirements is determined according to the roundness of the assembly hole.

[0026] In a possible implementation manner of the first aspect, the method further includes:

[0027] When the flatness of the assembly surface does not meet the first assembly requirement or the roundness of the assembly hole does not meet the second assembly requirement, a movement instruction and / or a prompt alarm is issued, wherein the movement instruction is used to instruct the assembly object to be moved to a cache station.

[0028] In a possible implementation manner of the first aspect, after controlling the fastener assembly device to implant the fastener into the assembly hole according to the 3D pose data of the center of the assembly hole, the method further includes:

[0029] Detecting whether the fastener is implanted in the assembly hole includes:

[0030] Acquire a second depth map of the assembly object;

[0031] Converting the second depth map into point cloud data to obtain second point cloud data;

[0032] Determining 3D position data of the center of the fastener according to the second point cloud data;

[0033] Taking the center of the circle of the fastener / assembly hole as the center, extracting point cloud data in a second area around the assembly hole from the second point cloud data; the second area is larger than or equal to the assembly hole area;

[0034] If the amount of point cloud data in the second area is within a preset threshold range, determining that the fastener is implanted in the assembly hole;

[0035] If the amount of point cloud data in the second area is not within a preset threshold range, it is determined that the fastener is not implanted in the assembly hole.

[0036] In a possible implementation manner of the first aspect, after detecting whether the fastener is implanted in the assembly hole, the method further includes:

[0037] Detecting whether the fastener is properly assembled includes:

[0038] Fitting a point cloud plane within the second region, and determining whether the point cloud within the second region is an interior point or an exterior point according to the point cloud plane within the second region and a preset second interior point threshold;

[0039] Determining the flatness of the point cloud in the second area according to the distribution of the point cloud in the second area as inner points and outer points;

[0040] Whether the fastener is assembled in place is determined according to the flatness of the point cloud in the second area.

[0041] In a possible implementation manner of the first aspect, after detecting whether the fastener is implanted in the assembly hole, the method further includes:

[0042] Detecting whether the fastener is properly assembled includes:

[0043] Determining the depth of the fastener implanted into the assembly hole according to the 3D pose data of the center of the assembly hole and the 3D pose data of the center of the fastener;

[0044] Whether the fastener is assembled in place is determined according to the depth of the fastener implanted into the assembly hole.

[0045] In a second aspect, an embodiment of the present application provides a fastener assembly system, including: a host computer, a depth camera, a control device, and a fastener assembly device;

[0046] The host computer is used to send an assembly request instruction and specification data and / or position data of the fastener to the fastener assembly device when determining that the assembly object meets the assembly conditions, and send an assembly request instruction to the depth camera;

[0047] The depth camera is used to collect a first depth map of the assembly object in response to the assembly request instruction; and send the first depth map to the control device, where the first depth map includes an assembly hole area of ​​the assembly object;

[0048] The control device is used to execute the method described in any one of the first aspects.

[0049] In a third aspect, an embodiment of the present application provides a fastener assembly device, the device comprising:

[0050] The communication module is used to receive assembly request instructions and specification data and / or position data of the fastener.

[0051] A first control module is used to obtain a first depth map of the assembly object in response to an assembly request instruction; and to grab the fastener through a fastener assembly device according to the specification data and / or position data of the fastener; the first depth map includes an assembly hole area of ​​the assembly object;

[0052] A positioning module, used to determine 3D pose data of the center of the assembly hole according to the first depth map;

[0053] The second control module is used to control the fastener assembly equipment to implant the fastener into the assembly hole of the assembly object according to the 3D position data of the center of the assembly hole.

[0054] In a fourth aspect, an embodiment of the present application provides a fastener assembly device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method as described in any one of the first aspects when executing the computer program.

[0055] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the first aspects is implemented.

[0056] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a fastener assembly device, the fastener assembly device executes any one of the methods in the first aspect.

[0057] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application can realize that the fastener assembly equipment automatically grasps the fastener, and automatically identifies the 3D position data of the center of the assembly hole, and then implants the fastener into the assembly hole, thereby realizing the automatic assembly of the fastener. The degree of automation is high, and it can greatly reduce the intensity of manual labor, improve production efficiency, and greatly improve the assembly quality of the fastener, which is not dependent on and limited by the experience of the operator, and reduces the defective rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 is a structural schematic diagram of a fastener assembly system provided in one embodiment of the present application;

[0061] Figure 2 is a schematic flow chart of a fastener assembly method provided in one embodiment of the present application;

[0062] Figure 3 It is a schematic diagram of a process for determining 3D pose data of the center of an assembly hole according to a first depth map provided by an embodiment of the present application;

[0063] Figure 4It is a schematic diagram of a flow chart of detecting whether the flatness of the assembly surface and the roundness of the assembly hole of the assembly object meet the assembly requirements provided by an embodiment of the present application;

[0064] Figure 5 It is a schematic diagram of a process for detecting whether a fastener is implanted in an assembly hole provided by an embodiment of the present application;

[0065] Figure 6 It is a point cloud diagram of the main process of image processing for positioning the plug assembly hole provided by an embodiment of the present application; Figure 6 Includes (a)-(d); Figure 6 (a) is a schematic diagram of a first depth map acquired by a depth camera; Figure 6 (b) is a schematic diagram of the point cloud data of the ROI area in the first point cloud data; Figure 6 (c) is a schematic diagram of the point cloud data after downsampling, filtering, clustering, and merging; Figure 6 (d) Schematic diagram of the circle detection algorithm determining the position of the center of the assembly hole;

[0066] Figure 7 is a point cloud schematic diagram of a first area provided in an embodiment of the present application;

[0067] Figure 8 is a point cloud schematic diagram of the main process of image processing for detecting whether a plug is inserted into a plug assembly hole provided in an embodiment of the present application; Figure 8 Includes (a)-(c); Figure 8 (a) Schematic diagram of a second depth map acquired by a depth camera; Figure 8 (b) is a schematic diagram of the point cloud data of the ROI area in the second point cloud data; Figure 8 (c) is a schematic diagram of the point cloud of the second area;

[0068] Fig. 9 It is a schematic diagram of the structure of the fastener assembly device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0070] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0071] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0072] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0073] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0074] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0075] Figure 1 The structural schematic diagram of the fastener assembly system provided by the present application is shown, and the system includes: a host computer 500, a depth camera 400, a control device 600, and a fastener assembly device 300. The control device 600 is used to control the fastener assembly device 300, so that the fastener assembly device 300 grabs the fastener 200, and the fastener assembly device 300 implants the fastener 200 into the assembly hole 101.

[0076] The control device 600 may be a processor in the fastener assembly device 300 or a control device independent of the fastener assembly device 300 , or the control device 600 may be the fastener assembly device 300 , which is not limited in the embodiments of the present application.

[0077] The host computer 500 is used to send an assembly request instruction and specification data and / or position data of the fastener 200 to the control device 600 and send an assembly request instruction to the depth camera 400 when determining that the assembly object meets the assembly conditions.

[0078] The depth camera 400 is used to collect a first depth map of the assembly object 100 in response to an assembly request instruction; the first depth map includes an assembly hole 101 area of ​​the assembly object 100 .

[0079] The control device 600 is used to receive an assembly request instruction and specification data and / or position data of a fastener 200; in response to the assembly request instruction, control the fastener assembly device 300 to grab the corresponding fastener 200; determine the 3D pose data of the center of the assembly hole 101 according to the first depth map; and control the fastener assembly device 300 to implant the fastener 200 into the assembly hole 101 of the assembly object 100 according to the 3D pose data of the center of the assembly hole 101.

[0080] As an example, the assembly object 100 may be a paper roll, and correspondingly, the fastener 200 is a plug, the assembly hole 101 is the plug assembly hole in the middle of the paper roll, and the fastener assembly device 300 is a plug assembly device. As another example, the assembly object 100 may be a red wine bottle, the fastener 200 is a wooden cork of the red wine bottle, the assembly hole 101 is the bottle mouth of the red wine bottle, and the fastener assembly device 300 is a plug assembly device. As another example, the assembly object 100 may be a component to be fixed, and correspondingly, the fastener 200 is a screw / screw, the assembly hole 101 is a threaded hole, and the fastener assembly device 300 is a screw / screw assembly device (such as an automatic screw tightening machine, an automatic screw locking machine, or a screw locking robot). As another example, the assembly object 100 may be a bearing, and correspondingly, the fastener 200 is a pin shaft, the assembly hole 101 is the mounting hole of the inner ring of the bearing, and the fastener assembly device 300 is a pin shaft press-fitting device. It is easy to understand that the fastener is a rod-shaped structure (the plug is truncated cone-shaped and can also be considered as a rod-shaped structure), so the method of the embodiment of the present application can be applied to the environment of rod-hole assembly.

[0081] The host computer 500 is a computer or system for monitoring, controlling and managing the assembly production line. The host computer 500 can be understood as an industrial computer. There is a communication protocol between the host computer 500, the control device 600 and the depth camera 400. There is a communication protocol between the control device 600 and the depth camera 400. Exemplarily, the assembly object 100 takes a paper roll as an example. In the paper roll-plug assembly production line, after the sensor detects that the paper roll has reached the assembly station, it sends a command to the host computer 500. After receiving the command sent by the sensor, the host computer 500 sends an assembly request command and the specification data and position data of the plug to the control device 600, and sends an assembly request command to the depth camera 400.

[0082] Exemplarily, the fastener assembly device 300 of the embodiment of the present application at least has a grasping mechanism (such as a manipulator, a clamp, a pneumatic finger, etc.) for grasping the fastener 200, a moving mechanism (such as a robot, a mechanical arm, a linear module, etc.) for moving the grasping mechanism, and an assembly mechanism (such as a press-fitting mechanism, a knocking mechanism, a tightening mechanism, etc.) for implanting the fastener 200 into the assembly hole 101. The grasping mechanism can grasp the outer peripheral surface of the fastener 200 (such as Figure 1 As shown), the mounting hole 201 in the axial direction of the fastener 200 can also be grasped. According to different grasping positions, the grasping mechanism is selected accordingly. Since the fastener assembly device 300 is a mature prior art, the specific structure of the fastener assembly device 300 is not described in detail in this embodiment.

[0083] The depth camera 400 is located on one side of the assembly surface 102 of the assembly object 100, and is used to collect a depth map (i.e., a first depth map) of the assembly surface 102 area of ​​the assembly object 100. Exemplarily, the depth camera 400 can be installed on the fastener assembly device 300 through a robotic arm or a linear module, or can be an independent device that is independent of the fastener assembly device 300 and can move, or can be an independent device that is fixed to one side of the assembly surface 102 and is stationary, and this embodiment does not limit this. It should be noted that when the depth camera 400 collects the first depth map, it can be shot directly facing the assembly hole 101 or tilted at a certain angle, but it is necessary to ensure that the first depth map includes the assembly hole 101 area and the assembly surface 102 in a certain area around the assembly hole 101.

[0084] The 3D pose data of the center of the assembly hole 101 refers to the position and direction of the center of the assembly hole 101 in three-dimensional space. The position is represented by (x, y, z), and the direction is described using a rotation matrix, Euler angles, or quaternions.

[0085] The fastener assembly system of this embodiment can realize that the fastener assembly device 300 automatically grabs the fastener 200, and automatically identifies the 3D pose data of the center of the assembly hole 101, and then implants the fastener 200 into the assembly hole 101, thereby realizing the automatic assembly of the fastener 200, with a high degree of automation. Through this embodiment, the labor intensity of manual labor can be greatly reduced, the production efficiency can be improved, the assembly quality of the fastener 200 can be greatly improved, it is not dependent on and limited by the experience of the operator, and the defective rate is reduced.

[0086] Figure 2 A schematic flow chart of the fastener assembly method provided by the present application is shown. As an example but not a limitation, the method can be applied to a fastener assembly device, which can be a fastener assembly device, or a control device for controlling the fastener assembly device, such as a chip or processor in the fastener assembly device, or a control device independent of the fastener assembly device, which is not limited in the present application embodiment. The method includes S110-S140.

[0087] S110: receiving an assembly request instruction and position data and / or specification data of a fastener.

[0088] Exemplarily, taking a paper roll as an assembly object, after the paper roll is transported to the assembly station by a transport device (such as a conveyor belt) and enters a ready state, the host computer sends an assembly request instruction to the processor of the plug assembly device, and the processor of the plug assembly device receives the assembly request instruction.

[0089] As an example, the processor of the fastener assembly device receives the position data and specification data of the fastener, and determines the position of the corresponding fastener according to the position data and specification data. As another example, the processor of the fastener assembly device stores the position data of different fastener specifications, so the position of the fastener can be determined by receiving the assembly request instruction and the specification data of the fastener. As another example, the fastener has only one specification, in which case the processor of the fastener assembly device only needs to receive the position data of the fastener, and determines the position of the fastener according to the position data.

[0090] The specifications of the fastener include the size and / or material of the fastener. Exemplarily, the fastener takes the plug of a paper roll as an example. The specifications of the plug include the size of the plug and the material of the plug. The material of the plug includes wood and plastic. The diameter of the plug includes 3 inches, 4 inches, 6 inches, 12 inches, etc. Different paper roll specifications correspond to different plug specifications, and plugs of different specifications are placed in different positions. Therefore, this embodiment can grab the plug of the corresponding specification according to the specifications of the paper roll.

[0091] S120: In response to an assembly request instruction, a first depth map of the assembly object is acquired, and corresponding fasteners are grabbed by a fastener assembly device according to specification data and / or position data of the fasteners; the first depth map includes an assembly hole area.

[0092] A depth map is an image that represents the distance between each pixel in a scene and a depth camera. In a depth map, each pixel value represents the distance of the point relative to the depth camera, measured in meters, millimeters, or other units.

[0093] Exemplarily, taking a plug assembly device as an example of a fastener assembly device, the processor of the plug assembly device responds to the assembly request instruction, controls the plug assembly device to grab the corresponding plug according to the specification and position data of the plug, and obtains the first depth map of the paper roll captured by the depth camera.

[0094] S130: Determine 3D pose data of the center of the assembly hole according to the first depth map.

[0095] S140: Controlling the fastener assembly device to implant the fastener into the assembly hole of the assembly object according to the 3D pose data of the center of the assembly hole.

[0096] Exemplarily, taking the plug assembly equipment as an example of the fastener assembly equipment, the processor of the plug assembly equipment parses the 3D pose data of the center of the assembly hole into a robot arm movement instruction, guides the robot arm to move the plug to the assembly hole, and the knocking mechanism knocks the plug into the assembly hole to complete the plug assembly operation.

[0097] The embodiment of the present application implants the fastener into the assembly hole through the fastener assembly device, automatically completes the fastener assembly work, has a high degree of automation, and greatly reduces the labor intensity of manual assembly. By detecting whether the flatness of the assembly surface of the assembly object and the roundness of the assembly hole meet the assembly requirements, the assembly quality of the fastener is effectively improved.

[0098] In an alternative embodiment, see Figure 3 S130: Determine the 3D pose data of the center of the assembly hole according to the first depth map, which specifically includes the following steps:

[0099] S131: Convert the first depth map into point cloud data to obtain first point cloud data.

[0100] Point Cloud is a collection of points in space that represent sampling points on the surface of an object.

[0101] Converting a depth map into point cloud data includes reading the depth image, extracting depth information, setting camera parameters, performing coordinate transformation, data filtering and optimization, and saving point cloud data.

[0102] S132: Determine the assembly surface point cloud data within the ROI area according to the first point cloud data.

[0103] The ROI area, namely the region of interest, is the area of ​​the assembly hole and the assembly surface within a certain range around the assembly hole.

[0104] Exemplarily, taking a paper roll as an assembly object, in order to ensure that the first depth map can cover the assembly hole area and the assembly surface area within a certain range around the assembly hole, the depth map range collected by the depth camera is relatively large and may include parts outside the paper roll, while the ROI area is only the assembly hole and the assembly surface area within a certain range around the assembly hole. Therefore, only the point cloud data within the ROI area needs to be extracted from the first point cloud data to reduce the amount of calculation and make the result more accurate.

[0105] Exemplarily, the extraction of the ROI region may be achieved in the following manner: performing a first preprocessing on the point cloud data within the ROI region to extract the assembly surface point cloud data within the ROI region; the first preprocessing includes downsampling, filtering, clustering, and merging.

[0106] Specifically, downsampling aims to reduce the amount of data and retain key information to improve computing efficiency and reduce storage requirements; by downsampling, the number of points can be significantly reduced while maintaining the basic form and characteristics of the point cloud, thereby reducing the complexity of the data and the computational burden. Point cloud data often contains unnecessary information such as noise points and outliers. These noise points not only affect the accuracy of subsequent processing, but may also interfere with the normal operation of the algorithm.

[0107] Through filtering algorithms (such as Gaussian filtering, statistical filtering, etc.), the influence of noise points can be removed or weakened, the quality of point cloud data can be improved, and a more reliable data basis can be provided for subsequent point cloud clustering, merging and other processing steps.

[0108] In point cloud data processing, clustering algorithms (such as K-means, DBSCAN, etc.) can discover the intrinsic structure in the point cloud, such as objects, surfaces or regions. Therefore, by clustering for feature extraction, the point cloud data can be divided into multiple parts with similar spatial positions, shapes or attributes, so as to better understand the data and make predictions. For example, the assembly object takes a paper roll as an example. Since the assembly hole and the surrounding assembly surface have significant differences in geometric features (such as curvature, normal vector, etc.), the assembly surface can be identified by analyzing these features. Curvature analysis: Calculate the local curvature of each point in the point cloud. When the first depth map collected by the depth camera includes the assembly surface, whether it is shot from the front or tilted, the inner wall of the assembly hole can be seen. The points on the inner wall of the through hole usually have higher curvature values, while the points on the plane have lower curvature. Normal vector analysis: As mentioned above, calculate the normal vector of each point in the point cloud. The normal vector of the inner wall of the assembly hole points in different directions, while the normal vectors on the plane are roughly the same.

[0109] Point cloud merging refers to the process of fusing multiple point cloud data sets into a larger and more complete point cloud. Through the point cloud merging algorithm, point cloud data sets with different characteristics can be distinguished, thereby removing redundant and inconsistent parts to generate a unified and complete point cloud model. After obtaining the point cloud model, the point cloud data of the assembly surface is extracted based on the similarity of the normal vector, thereby obtaining the assembly surface point cloud data within the ROI area.

[0110] S133: Projecting the assembly surface point cloud data within the ROI region into a depth map to obtain a second depth map.

[0111] The process of projecting planar point cloud data into a depth map is to map points in three-dimensional space onto a two-dimensional plane and encode the depth information of these points into pixel values ​​in the image. This involves the following steps: determining the projection plane and coordinate system, extracting point cloud data, calculating depth values, mapping to the image plane, normalizing depth values, creating a depth map, saving and displaying the depth map, etc.

[0112] S134: Determine the position of the center of the assembly hole by using a circle detection algorithm according to the second depth map.

[0113] The second depth map is processed by denoising and filling, and the position of the center of the assembly hole and the radius of the assembly hole can be determined by the circle detection algorithm. Denoising is to use algorithms and models to eliminate or reduce noise in the data to improve the quality and availability of the data. Filling is to fill the holes or sparse areas in the second depth map to complete these defective information and make the second depth map more complete and continuous.

[0114] The position of the center of the assembly hole refers to the coordinates (x, y values) of the center in two-dimensional space.

[0115] Specifically, the circle detection algorithm includes but is not limited to Hough Transform, template matching method based on edge detection, gradient method, support vector machine method (SVM), random sample consensus algorithm (RANSAC), circular Hough gradient method (Circular Hough Transform with Gradient), least square method, etc. The center position and radius of the assembly hole can be obtained through the circle detection algorithm.

[0116] S135: Projecting the position of the center of the assembly hole into 3D pose data of the center of the assembly hole.

[0117] Exemplarily, the position of the center of the assembly hole is converted back to the 3D pose data of the center of the assembly hole based on the projection relationship in S134.

[0118] The 3D pose data of the center of the assembly hole refers to the position and direction of the center of the assembly hole in three-dimensional space. The center position is represented by (x, y, z), and the center direction is described using a rotation matrix, Euler angle, or quaternion.

[0119] This embodiment converts the first depth map into the first point cloud data, and then extracts the assembly surface point cloud data of the ROI area in the first point cloud data. In order to facilitate the measurement of the center of the assembly hole, the extracted assembly surface point cloud data is projected into a second depth map, the second depth map is denoised and filled, the position of the fastener is calculated by the circle detection algorithm, and then the position of the center of the assembly hole is converted back to the 3D pose data of the center of the assembly hole based on the previous projection relationship. This embodiment can quickly obtain the 3D pose data of the center of the assembly hole based on the first depth map, and the assembly hole positioning accuracy is high.

[0120] As an optional implementation, before S140, the fastener assembly method of the embodiment of the present application further includes S150.

[0121] S150: Detect the flatness of the assembly surface and the roundness of the assembly hole of the assembly object.

[0122] The assembly face is the end face of the assembly object facing the fastener. For example, see Figure 1 , the assembly object takes a paper roll as an example. The paper roll is a circular cylinder. The circular cylinder is a spatial geometric body surrounded by two parallel and coaxial circles, and its cross section is a circular ring. The paper roll has two approximately flat circular ring end faces. In order to increase the pressure resistance of the paper roll, a paper core is set in the center of the paper roll, and the paper core cavity is the assembly hole. The assembly surface of the paper roll is a circular ring end face facing the plug and the depth camera.

[0123] Flatness is an indicator of whether a surface is close to an ideal plane state. Exemplarily, taking a paper roll as an assembly object, the flatness of the assembly surface refers to the degree to which the assembly surface of the paper roll is close to an ideal plane. It is easy to understand that the assembly surface of the paper roll is only approximately a plane. The smaller the flatness of the assembly surface of the paper roll, the flatter the assembly surface; the greater the flatness, the more uneven the assembly surface. In the process of implanting the plug into the assembly hole of the paper roll, it is necessary to ensure that the assembly surface is relatively flat. For example, if the paper roll is spindle-shaped, it means that the flatness of the assembly surface is poor. If the assembly hole is partially protruding, the plug will be implanted too deep into the assembly hole, causing damage to the plug or the assembly hole; if the assembly hole is partially concave, the plug will not be implanted into the assembly hole properly, causing the plug to fall off easily.

[0124] Roundness is used to evaluate the degree of deviation between a circle and an ideal circle. The roundness of an assembly hole refers to the degree of deviation between the assembly hole and the ideal circle. For example, the assembly object is a paper roll. The assembly hole of the paper roll is a circle. When there is no external force pressing the paper roll or the assembly hole is deformed due to the deadweight of the paper roll, the assembly hole has a high roundness. If the paper roll is squeezed by an external force or the assembly hole is deformed into an ellipse due to its deadweight, the assembly hole has a low roundness. In the process of inserting the plug into the assembly hole of the paper roll, it is necessary to ensure that the assembly hole has a high roundness. Otherwise, if the assembly hole becomes an ellipse, the plug may not be able to be inserted into the assembly hole.

[0125] S150 specifically includes: if the flatness of the assembly surface of the assembly object meets the first assembly requirement and the roundness of the assembly hole meets the second assembly requirement, then according to the 3D pose data of the center of the assembly hole, controlling the fastener assembly equipment to implant the fastener into the assembly hole.

[0126] The first assembly requirement is that the flatness of the assembly surface meets the preset flatness of the assembly surface. The second assembly requirement is that the roundness of the assembly hole meets the preset roundness of the assembly hole.

[0127] As an optional implementation, see Figure 4 , S150: Detect whether the flatness of the assembly surface and the roundness of the assembly hole of the assembly object meet the assembly requirements, specifically including S151-S155.

[0128] S151: Taking the center of the assembly hole as the center, extracting point cloud data of a first area around the assembly hole from the first point cloud data; the first area is larger than the assembly hole area and smaller than the ROI area.

[0129] In order to further reduce the amount of calculation, the ROI area is further reduced. Since the 3D pose data of the center of the assembly hole is already known, the first area can be determined with the center of the assembly hole as the center and a radius greater than the assembly hole as the radius.

[0130] When the depth camera of the embodiment of the present application collects the first depth map, since the depth camera is not sure of the position of the assembly hole, the area collected by the depth camera is relatively large to ensure that the first depth map can cover the position of the assembly hole. After the first depth map is converted into point cloud data, the ROI area can be extracted according to the characteristics of the point cloud data. The ROI area greatly reduces the amount of point cloud data and can cover the assembly hole area. After determining the 3D pose data of the center of the assembly hole, the first area is determined according to the center and radius of the assembly hole, which can further reduce the point cloud data. It should be noted that if the first area is circular, the radius of the first area is greater than the radius of the assembly hole, that is, in addition to the assembly hole area, the first area also includes the plane area around the assembly hole.

[0131] Exemplarily, the first area may be a circular area covering the assembly hole, and the center in this case may be understood as the center of the circle. It is easy to understand that the second area may also be other shapes.

[0132] S152: Fitting a point cloud plane within the first region, and determining whether a point cloud within the first region is an inner point or an outer point according to the point cloud plane within the first region and a preset first inner point threshold.

[0133] Furthermore, before determining whether the point cloud in the first region is an inner point or an outer point, outliers are first removed by filtering. Outliers are points that are obviously not on the assembly surface, and whether they are outliers can be determined by the distance between the point cloud and the point cloud plane in the fitted first region.

[0134] It is easy to understand that the assembly surface is only approximated to a plane, and the point clouds in the first area are not all on a certain plane, but are distributed at a certain thickness. Therefore, in order to determine the distribution of the point clouds in the first area, you can first fit a point cloud plane (which is an ideal plane / decision plane), and then set a suitable first inner point threshold (threshold about distance) to determine the inner point range. The inner point range refers to the thickness range between the upper and lower first inner point thresholds of the point cloud plane. The point clouds within the inner point range are determined as inner points, and the rest are determined as outer points, and then the distribution of inner points and outer points can be determined.

[0135] For example, the RANDOM SAmple Consensus algorithm can be used to fit the point cloud plane in the first region. The model is fitted by randomly selecting a small number of data points (minimum sample set) and calculating the number of inliers in the data set that conform to the model. Through multiple iterations, the RANSAC algorithm can find the optimal model parameters to maximize the number of inliers.

[0136] S153: Determine the flatness of the assembly surface according to the distribution of the point cloud within the first area at the inner points and the outer points.

[0137] Flatness refers to the difference between the assembly surface and the absolute level. This difference reflects the unevenness of the assembly surface. The smaller the value, the flatter the surface.

[0138] S154: Determine whether the assembly surface meets assembly requirements based on the flatness of the assembly surface.

[0139] As an example, whether the flatness of the assembly surface meets the assembly requirements is determined by the flatness error. The flatness error is a statistic of the distance from all the inner points in the point cloud (i.e., points whose distance from the fitting plane is less than the first inner point threshold) to the fitting plane, such as the root mean square value (RMS) or other parameters describing the distribution. When the flatness error of the point cloud in the first area is less than the preset flatness error, it is determined that the flatness of the assembly surface meets the assembly requirements. When the flatness error of the point cloud in the first area is greater than the preset flatness error, it is determined that the flatness of the assembly surface does not meet the assembly requirements.

[0140] It is easy to understand that the more point clouds are located in the inner point range, the better the flatness of the point clouds in the first region. Conversely, the fewer point clouds are located in the inner point range, the worse the flatness of the point clouds in the first region.

[0141] As another example, whether the flatness of the assembly surface meets the assembly requirements can be determined based on the proportion of the internal points to the total point cloud. For example, if more than 95% of the point cloud is an internal point, it is determined that the flatness of the assembly surface meets the assembly requirements.

[0142] S155: Detect the roundness of the assembly hole, and determine whether the assembly hole meets the assembly requirements based on the roundness of the assembly hole.

[0143] Roundness refers to the degree to which a mounting hole differs from a perfect circle.

[0144] The roundness of the assembly hole is detected by a roundness detection algorithm, which is used to determine the deviation between the assembly hole and the ideal circle. The roundness detection algorithm can use the minimum area method, the least squares circle method, the minimum circumscribed circle method, the maximum inscribed circle method, the rotary axis method or the three-point method.

[0145] The embodiment of the present application detects whether the flatness of the assembly surface and the roundness of the assembly hole of the assembly object meet the assembly requirements, and only performs S140 when the flatness of the assembly surface and the roundness of the assembly hole meet the assembly requirements, thereby greatly improving the assembly quality. It reduces the situation where the assembly fails or the assembly object is damaged due to the flatness of the assembly surface and the roundness of the assembly hole not meeting the assembly requirements.

[0146] As an optional implementation, the fastener assembly method of the embodiment of the present application further includes S160.

[0147] S160: When the flatness of the assembly surface does not meet the first assembly requirement or the roundness of the assembly hole does not meet the second assembly requirement, a movement instruction and / or a prompt alarm is issued, where the movement instruction is used to instruct to move the assembly object to a cache station.

[0148] For example, if it is detected that the flatness of the assembly surface or the roundness of the assembly hole does not meet the assembly requirements, the processor of the fastener assembly equipment sends a movement instruction to the host computer, and the host computer instructs the conveying device (such as a conveyor belt) to move the assembly object to the cache station for manual processing according to the movement instruction; or, the assembly object is moved to the cache station manually for manual processing; or, the rejection mechanism in the assembly production line moves the assembly object to the cache station according to the removal instruction.

[0149] As an optional implementation, after S140: controlling the fastener assembly device to implant the fastener into the assembly hole of the assembly object according to the 3D pose data of the center of the assembly hole, the fastener assembly method of the embodiment of the present application further includes S170.

[0150] S170: Check whether the fastener is inserted into the assembly hole. Figure 5 Specifically, S170 includes S171-S175.

[0151] S171: Acquire a second depth map of the assembly object.

[0152] The depth camera collects a depth map of the assembled object again to obtain a second depth map.

[0153] S172: Convert the second depth map into point cloud data to obtain second point cloud data.

[0154] Optionally, S172 may obtain the second point cloud data according to the method of S131, and therefore, this embodiment will not be described in detail herein.

[0155] S173: Determine 3D position data of the center of the fastener according to the second point cloud data.

[0156] Optionally, S173 can determine the 3D pose data of the center of the fastener according to the method of S132-S135. Specifically, it includes determining the point cloud data of the assembly surface in the ROI area from the second point cloud data; projecting the point cloud data of the assembly surface in the ROI area into a depth map to obtain a third depth map; determining the position of the center of the fastener through a circle detection algorithm according to the third depth map; and projecting the position of the center of the fastener into the 3D pose data of the center of the fastener.

[0157] S174: Taking the center of the circle of the fastener / assembly hole as the center, extracting point cloud data in a second area around the assembly hole from the second point cloud data; the second area is larger than or equal to the assembly hole area.

[0158] For example, the plug is taken as an example of a fastener, and the point cloud data in the second area is extracted with the center of the plug as the center and the radius of the plug as the radius. The radius of the plug can be obtained from the specification of the plug or from a circle detection algorithm.

[0159] The second area may be circular or in other shapes.

[0160] S175: If the amount of point cloud data in the second area is within a preset threshold range, determine that the fastener is implanted in the assembly hole.

[0161] S176: If the amount of point cloud data in the second area is not within a preset threshold range, it is determined that the fastener is not implanted in the assembly hole.

[0162] It is easy to understand that before the fastener is implanted in the assembly hole, the number of point clouds is small, and after the fastener is implanted in the assembly hole, the number of point clouds increases. Based on this, the number of point cloud data in the second area can be compared with the preset threshold range to determine whether the fastener is implanted in the assembly hole.

[0163] This embodiment can quickly detect whether the fastener is implanted in the assembly hole without adding any hardware equipment, and greatly improves the assembly quality.

[0164] As an optional implementation, after S170: detecting whether the fastener is implanted in the assembly hole, the fastener assembly method of the embodiment of the present application further includes S180.

[0165] S180: Check whether the fastener is assembled in place. Specifically, S180 includes S181-S183.

[0166] S181: Fitting the point cloud plane in the second area, and determining whether the point cloud in the second area is an interior point or an exterior point according to the point cloud plane in the second area and a preset second interior point threshold.

[0167] Optionally, S181 may determine whether the point cloud in the second area is an inner point or an outer point according to the method of S152, and therefore, this embodiment will not be described in detail herein.

[0168] S182: Determine the flatness of the point cloud in the second area according to the distribution of the point cloud in the second area as inner points and outer points.

[0169] Optionally, the flatness of the point cloud in the second area may be reflected by a root mean square value or a proportion of the inner points in the total point cloud data.

[0170] S183: Determine whether the fastener is assembled in place according to the flatness of the point cloud in the second area.

[0171] As an example, if the flatness error of the point cloud in the second area is less than the preset flatness error, it is determined that the fastener is assembled in place. If the flatness error of the point cloud in the second area is greater than the preset flatness error, it is determined that the fastener is not assembled in place.

[0172] As another example, if the ratio of the inner points of the second region to the overall point cloud is greater than or equal to a set ratio, it is determined that the fastener is in place. If the ratio of the inner points of the second region to the overall point cloud is less than a set ratio, it is determined that the fastener is not in place.

[0173] The methods of S181-S183 are applicable to the case where the end face of the fastener is flush with the assembly surface after the fastener is assembled in place. For example, when the plug and the assembly hole of the paper roll are assembled in place, the end face of the plug is flush with the assembly surface of the paper roll. When the countersunk screw and the countersunk screw hole are assembled in place, the end face of the countersunk screw is flush with the assembly plane.

[0174] As another optional implementation, S180: detecting whether the fastener is assembled in place, specifically includes S184-S185.

[0175] S184: Determine the depth of the fastener implanted into the assembly hole according to the 3D pose data of the center of the assembly hole and the 3D pose data of the center of the fastener.

[0176] It is easy to understand that the 3D pose data of the center of the assembly hole contains the coordinates of the center of the assembly hole in three-dimensional space (X, Y, and Z dimensions), where the Z coordinate directly reflects the depth information (i.e., distance) from the center of the assembly hole to the depth camera. Similarly, the 3D pose data of the center of the fastener can also reflect the depth information from the fastener to the depth camera. The difference between the depth information from the assembly hole to the depth camera and the depth from the fastener to the depth camera can determine the depth of the fastener implanted in the assembly hole.

[0177] S185: Determine whether the fastener is properly assembled based on the depth of the fastener implanted into the assembly hole.

[0178] Methods S184-S185 are applicable to the case where the end face of the fastener is not in the same plane as the assembly surface after the fastener is assembled. For example, during the assembly process of the wine bottle stopper and the wine bottle assembly hole, after the wine bottle stopper is assembled, the end face of the wine bottle stopper is not flush with the wine bottle assembly surface (the end face at the wine bottle assembly hole). If the wine bottle stopper is inserted into the wine bottle assembly hole 2.8-3.2cm, it is considered to be assembled in place, and if it is outside this range, it can be determined that the wine bottle stopper is not assembled in place.

[0179] As an optional implementation, if it is detected that the fastener in S175 is not implanted in the assembly hole, and the fastener in S183 or S185 is not assembled in place, a movement instruction and / or a prompt alarm is issued, and the movement instruction is used to instruct the assembly object to be moved to the cache station.

[0180] Furthermore, the fastener assembly method of the embodiment of the present application also includes S190.

[0181] S190: If the fastener is implanted into the assembly hole and the fastener is assembled in place, the fastener assembly device is controlled to move to the preparation position and an assembly completion signal is sent.

[0182] After detecting whether the fastener is implanted in the assembly hole, this embodiment further detects whether the fastener is assembled in place, thereby greatly improving the assembly quality of the fastener.

[0183] This implementation takes the assembly of a 4-inch plug and a paper roll as an example to illustrate the plug assembly method:

[0184] Step 1: The processor of the plug assembly device receives a plug assembly request instruction sent by the host computer, and the plug assembly request instruction includes plug specification data and position data.

[0185] The plug assembly request signal received by the processor of the plug assembly device from the host computer system is 1, indicating that the material has arrived at the assembly station and is ready to be assembled with the plug. The plug specification includes the size and material of the plug. The plug size data is [4,20], which means that the plug has a diameter of 4 inches and a thickness of 20 mm. The material of the plug is wood.

[0186] Step 2: Control the plug assembly equipment to grab the plug and move the depth camera to the visual inspection position.

[0187] Step 3: Locate the plug assembly hole.

[0188] Figure 6 This is a point cloud diagram of the main image processing process in the positioning plug assembly hole. Figure 6 include Figure 6 (a)- Figure 6 (d). Figure 6 (a) is a schematic diagram of a first depth map acquired by a depth camera; Figure 6 (b) is a schematic diagram of the point cloud data of the ROI area in the first point cloud data; Figure 6 (c) is a schematic diagram of the point cloud data after downsampling, filtering, clustering, and merging; Figure 6 (d) Schematic diagram of the circle detection algorithm determining the location of the center of the assembly hole.

[0189] The first depth map is converted into point cloud data, and after downsampling, filtering, clustering, and merging, the plane point cloud data of the assembly surface in the ROI area is extracted. To facilitate the measurement of the center of the assembly hole, the extracted assembly surface point cloud is projected into a second depth map, and the second depth map is denoised and filled. The position of the center of the plug assembly hole is calculated by the circle detection algorithm, and then the position of the center of the plug assembly hole is converted back to the 3D pose data of the center of the plug assembly hole based on the previous projection relationship.

[0190] The center position of the plug assembly hole is {"theta": 0,"x": 532.107177734375,"y":543.2019653320312};

[0191] Theta represents the direction of the figure, and x and y represent the coordinates of the center of the circle in the two-dimensional coordinate system.

[0192] 3D pose data of the center of the plug assembly hole {"trans": [1.9097649519976647,-0.7178604446238943,0.13068929304749655], "rot":[-0.7825572440630738,-0.02484925721560618,-0.018347880634225367,-0.6218118923408201]};

[0193] Where trans represents the coordinates of the center of the circle in the three-dimensional coordinate system, and rot represents the quaternion that describes the posture of the center of the circle.

[0194] Step 4: Check the flatness of the plug assembly surface and the roundness of the plug assembly hole.

[0195] like Figure 7 As shown, Figure 7 The figure is a schematic diagram of the point cloud of the first area. The point cloud data of the first area is extracted with the center of the circle of the plug assembly hole as the center. After filtering to remove outliers, an appropriate first inner point threshold is set, and the point cloud plane is fitted using a random sampling consistency algorithm. The flatness error is calculated based on the distribution of the inner and outer point clouds of the fitting results. This is used as a basis to determine whether the flatness of the plug assembly plane meets the subsequent assembly requirements. Check whether the roundness of the plug assembly hole meets the subsequent assembly requirements.

[0196] The measured flatness error is: 0.00271451473236084.

[0197] Step 5: According to the 3D pose data of the center of the assembly hole, guide the plug assembly equipment to knock the plug into the assembly hole.

[0198] The processor of the plug assembly equipment sends instruction data format: ;

[0199] The plug assembly hole data format obtained by the processor of the plug assembly device is:

[0200] ;

[0201] Visual point 1 data structure: ;

[0202] In the above table, visual point 1 represents the 3D pose data of the center of the plug assembly hole.

[0203] Step 6: Check the plug assembly quality.

[0204] Figure 8 A point cloud diagram of the main image processing process for detecting whether the plug is inserted into the plug assembly hole. Figure 8 include Figure 8 (a)- Figure 8 (c). Figure 8 (a) Schematic diagram of the second depth map acquired by the depth camera. Figure 8 (b) is a schematic diagram of the point cloud data of the ROI area in the second point cloud data. Figure 8 (c) is a schematic diagram of the point cloud of the second area.

[0205] Detect whether the plug is implanted in the plug assembly hole: take the coordinates of the center position of the plug as the center of the circle, set the radius according to the size of the plug, extract the point cloud data in the second area, and if the number of point cloud points is within the threshold range, it is considered that there is a plug in the plug assembly hole, otherwise it is considered that the plug is not assembled in the plug assembly hole.

[0206] Test result: true (success) / false (failure).

[0207] Check whether the plug is properly assembled. Figure 8 (c) The flatness error of the second area point cloud is calculated.

[0208] The outliers in the second area are removed by filtering. An appropriate second inlier threshold is set, and the point cloud plane is fitted using a random sampling consistency algorithm. The flatness error is calculated based on the distribution of the inlier and outlier point clouds of the fitting results to determine whether there are bulges or depressions in the plug assembly hole and other inadequate assembly phenomena.

[0209] The measured flatness error is: 0.00271451473236084.

[0210] In step 4, if it is detected that the flatness of the assembly surface or the roundness of the plug assembly hole does not meet the assembly requirements, the paper roll will be moved to the buffer station for manual processing.

[0211] In step six, if it is detected that the plug is not inserted into the plug assembly hole or is not assembled in place, the paper roll will be moved to the buffer station for manual processing.

[0212] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0213] Corresponding to the fastener assembly method described in the above embodiment, Fig. 9 A structural block diagram of a fastener assembly device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0214] Reference Fig. 9 , the device comprises:

[0215] The communication module 910 is used to receive an assembly request instruction and specification data and / or position data of a fastener.

[0216] A first control module 920 is configured to obtain a first depth map of the assembly object in response to an assembly request instruction; and to grab the fastener through a fastener assembly device according to the specification data and / or position data of the fastener; the first depth map includes an assembly hole area of ​​the assembly object;

[0217] A positioning module 930 is used to determine 3D pose data of the center of the assembly hole according to the first depth map;

[0218] The second control module 940 is used to control the fastener assembly device to implant the fastener into the assembly hole of the assembly object according to the 3D position data of the center of the assembly hole.

[0219] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0220] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0221] An embodiment of the present application also provides a fastener assembly device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0222] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0223] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0224] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the fastener assembly equipment, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0225] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0226] The computer program code for performing the operation of the embodiment of the application can be written with one or more programming languages ​​or their combination, and the programming language includes object-oriented programming languages, such as python, Java, Smalltalk, C++, and also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or executed completely on the remote computer or server. In the case of a remote computer, the remote computer can include a local area network (LAN) or a wide area network (WAN)--connected to the user's computer through any type of network, or, can be connected to an external computer (for example, utilizing an Internet service provider to connect through the Internet).

[0227] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0228] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0229] In the embodiments provided in the present application, it should be understood that the disclosed device / fastener assembly equipment and method can be implemented in other ways. For example, the device / fastener assembly equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A fastener assembly method, characterized in that: The method comprises: receiving an assembly request instruction and specification data and / or location data of a fastener; In response to the assembly request instruction, acquiring a first depth map of the assembly object, and grabbing the fastener by a fastener assembly device according to the specification data and / or position data of the fastener; The first depth map includes the assembly hole area of ​​the assembly object, and the first depth map is converted into point cloud data to obtain first point cloud data; according to the first point cloud data, the assembly surface point cloud data in the ROI area is determined; the assembly surface point cloud data in the ROI area is projected into a depth map to obtain a second depth map; according to the second depth map, the position of the center of the assembly hole is determined by a circle detection algorithm; the position of the center of the assembly hole is projected into 3D pose data of the center of the assembly hole; Detect the flatness of the assembly surface of the assembly object and the roundness of the assembly hole, extract point cloud data of a first area around the assembly hole from the first point cloud data with the center of the assembly hole as the center; the first area is larger than the assembly hole area and smaller than the ROI area; fit the point cloud plane in the first area, and determine whether the point cloud in the first area is an inner point or an outer point according to the point cloud plane in the first area and a preset first inner point threshold; determine the flatness of the assembly surface according to the distribution of the point cloud in the first area as an inner point or an outer point; detect the roundness of the assembly hole by a roundness detection algorithm; If the flatness of the assembly surface of the assembly object meets the first assembly requirement and the roundness of the assembly hole meets the second assembly requirement, the fastener assembly device is controlled to implant the fastener into the assembly hole according to the 3D pose data of the center of the assembly hole.

2. The fastener assembly method according to claim 1, characterized in that: The method further comprises: When the flatness of the assembly surface does not meet the first assembly requirement or the roundness of the assembly hole does not meet the second assembly requirement, a movement instruction and / or a prompt alarm is issued, wherein the movement instruction is used to instruct the assembly object to be moved to a cache station.

3. The fastener assembly method according to any one of claims 1 to 2, characterized in that: Also includes: Detecting whether the fastener is implanted in the assembly hole includes: Acquire a second depth map of the assembly object; Converting the second depth map into point cloud data to obtain second point cloud data; Determining 3D position data of the center of the fastener according to the second point cloud data; Taking the center of the circle of the fastener / assembly hole as the center, extracting point cloud data in a second area around the assembly hole from the second point cloud data; the second area is larger than or equal to the assembly hole area; If the amount of point cloud data in the second area is within a preset threshold range, determining that the fastener is implanted in the assembly hole; If the amount of point cloud data in the second area is not within a preset threshold range, it is determined that the fastener is not implanted in the assembly hole.

4. The fastener assembly method according to claim 3, characterized in that: The method further comprises: Fitting a point cloud plane within the second region, and determining whether the point cloud within the second region is an interior point or an exterior point according to the point cloud plane within the second region and a preset second interior point threshold; Determining the flatness of the point cloud in the second area according to the distribution of the point cloud in the second area as inner points and outer points; Whether the fastener is assembled in place is determined according to the flatness of the point cloud in the second area.

5. The fastener assembly method according to claim 3, characterized in that: The method further comprises: Determining the depth of the fastener implanted into the assembly hole according to the 3D pose data of the center of the assembly hole and the 3D pose data of the center of the fastener; Whether the fastener is assembled in place is determined according to the depth of the fastener implanted into the assembly hole.

6. A fastener assembly system, characterized in that: include: Host computer, depth camera, control equipment and fastener assembly equipment; The host computer is used to send an assembly request instruction and specification data and / or position data of the fastener to the fastener assembly device when determining that the assembly object meets the assembly conditions, and to send an assembly request instruction to the depth camera respectively; The depth camera is used to collect a first depth map of the assembly object in response to the assembly request instruction, and send the first depth map to the control device, where the first depth map includes an assembly hole area of ​​the assembly object; The control device is used to execute the method according to any one of claims 1 to 5.

7. A fastener assembly device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Shaft hole assembling method based on three-dimensional point cloud

    CN118552592A

  • System and method for determining a flushness of a fastener

    US20200151893A1

  • System and method for autonomously removing fasteners embedded in wood products

    US20230281793A1

  • Method and system for measuring wear of workpiece plane

    WO2022105676A1