Robot autonomous tightening method and system based on three-dimensional point cloud key point detection and adaptive decision

Abstract

The application provides a robot autonomous tightening method and system based on three-dimensional point cloud key point detection and adaptive decision, and relates to the technical field of intelligent control of industrial robots. The method comprises: acquiring three-dimensional point cloud data of a work scene and segmenting target point cloud data of a target fastener; inputting the target point cloud data into a trained point cloud feature extraction network to regress and predict the spatial position of a predetermined key point; determining a six-degree-of-freedom pose according to the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on a standard model; determining the attitude category and the type of work to be performed according to the six-degree-of-freedom pose; when performing attitude adjustment work, controlling the robot end effector to perform attitude adjustment on the target fastener, and reacquiring point cloud data after adjustment to update the pose and the attitude category; when performing grabbing work, controlling the robot end effector to grab the target fastener and perform a tightening operation according to the six-degree-of-freedom pose.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology for industrial robots, specifically to a robot autonomous tightening method and system based on 3D point cloud key point detection and adaptive decision-making. Background Technology

[0002] The automated assembly of fasteners such as bolts and nuts is a key process in automobile manufacturing, aerospace, and precision electronics production.

[0003] In related technologies, 2D vision-based guidance techniques acquire planar images of parts using industrial cameras and obtain pose information through edge detection or template matching. However, they cannot provide depth information and struggle to handle arbitrary placement of fasteners in three-dimensional space. 3D point cloud matching methods obtain pose through point cloud registration, but due to the strong specular reflection of metal fastener surfaces, point cloud data often contains holes, noise, or artifacts. Traditional registration algorithms experience a sharp drop in accuracy on such poor-quality point clouds, making it difficult to meet the accuracy requirements for threaded mating. Furthermore, some solutions rely on six-dimensional force sensors for force control compensation, but their high cost limits large-scale deployment.

[0004] Therefore, how to achieve high-precision pose detection and autonomous assembly of reflective metal fasteners in any orientation under low-cost hardware conditions is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a robot autonomous tightening method and system based on 3D point cloud key point detection and adaptive decision-making, which can achieve high-precision pose detection and autonomous assembly of metal reflective fasteners in any orientation under low-cost hardware conditions.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making, the method comprising: Acquire 3D point cloud data of the work scene; The target point cloud data of the target fastener is segmented from the three-dimensional point cloud data; The target point cloud data is input into a trained point cloud feature extraction network to regress and predict the spatial location of predetermined key points on the target fastener. Based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener, the six-degree-of-freedom pose of the target fastener is determined; The attitude category of the target fastener is determined based on the six-degree-of-freedom pose, and the attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping; Based on the posture category, determine the type of task to be performed, which includes grasping tasks and posture adjustment tasks; When it is determined that the attitude adjustment operation should be performed, the robot end effector is controlled to perform the attitude adjustment operation on the target fastener, and the three-dimensional point cloud data of the target fastener is reacquired after the attitude adjustment operation to update the six-degree-of-freedom pose and the attitude category; When it is determined that the gripping operation should be performed, the robot end effector grips the target fastener according to the six-degree-of-freedom pose control and performs the tightening operation.

[0007] Optionally, segmenting the target point cloud data of the target fastener from the three-dimensional point cloud data includes: The three-dimensional point cloud data is spatially filtered according to preset spatial range parameters to filter out background point clouds outside the area where the target fastener is located. Outlier removal is performed on the filtered point cloud data; Clustering and segmentation of the point cloud data after filtering out outliers is performed based on the point cloud density distribution to obtain the target point cloud of the target fastener.

[0008] Optionally, outlier removal is performed on the filtered point cloud data, including: Simulated light rays are projected from a preset viewpoint onto each point in the filtered point cloud data; If there are other point cloud occlusions on the path connecting the current point and the preset viewpoint position, then the target point is retained; If there are no other point cloud obstructions on the path connecting the current point and the preset viewpoint, the target point is identified as an outlier and removed.

[0009] Optionally, the point cloud feature extraction network is trained according to the following steps: Construct a training dataset, wherein each training sample in the training dataset includes three-dimensional point cloud data of the sample fastener and corresponding annotation information, wherein the annotation information includes the true coordinates of multiple predetermined key points on the sample fastener and the true height of the sample fastener; The three-dimensional point cloud data of the sample fastener is input into the point cloud feature extraction network to be trained to obtain the predicted coordinates of the multiple predetermined key points and the predicted height of the sample fastener. Calculate the key point loss based on the difference between the predicted coordinates and the actual coordinates of the multiple predetermined key points; Based on the spatial collinearity relationship between the predicted coordinates of the multiple predetermined key points, the direction loss is calculated. Calculate the size loss based on the difference between the predicted height and the actual height; The network parameters of the point cloud feature extraction network to be trained are updated based on the key point loss, the orientation loss, and the size loss.

[0010] Optionally, the predetermined key points include a first key point, a second key point, and a third key point, wherein the first key point, the second key point, and the third key point correspond to the top feature position, the middle feature position, and the bottom feature position of the target fastener, respectively. Based on the transformation relationship between the spatial positions of the predetermined key points and the positions of the corresponding key points on the standard model of the target fastener, the six-degree-of-freedom pose of the target fastener is determined, including: The feature point type corresponding to each of the predetermined key points is determined according to the standard model, and the feature point type includes top feature point, middle feature point and bottom feature point; Based on the feature point type, determine the positions of the first key point, the second key point, and the third key point from the spatial positions of the predetermined key points; A first direction vector is determined based on the position of the first key point and the position of the second key point, and a second direction vector is determined based on the position of the second key point and the position of the third key point. Based on the average direction of the first direction vector and the second direction vector, the axial direction of the target fastener is determined, and the spatial position of the key point corresponding to the top feature point of the standard model among the predetermined key points is determined as the origin of the local coordinate system. A local coordinate system for the target fastener is constructed based on the axial direction and the origin, and the six-degree-of-freedom pose is determined based on the rotation and displacement relationships of the local coordinate system relative to the reference coordinate system; wherein, the reference coordinate system is the coordinate system in which the three-dimensional point cloud data is located.

[0011] Optionally, determining the attitude category of the target fastener based on the six-degree-of-freedom pose includes: The central axis direction vector of the target fastener is extracted from the six-degree-of-freedom pose, and the central axis direction vector points from the bottom end to the top end of the target fastener; Obtain the gravity direction vector and calculate the angle between the central axis direction vector and the gravity direction vector; When the included angle is less than or equal to the first preset angle threshold, the posture category is determined to be the first posture category, which indicates that the target fastener is in a posture with the top facing down. When the included angle is greater than the first preset angle threshold and less than the second preset angle threshold, the posture category is determined to be the second posture category; the second posture category indicates that the target fastener is in a side-lying posture, and the first preset angle threshold is less than the second preset angle threshold; When the included angle is greater than or equal to the second preset angle threshold, the posture category is determined to be the third posture category, which indicates that the target fastener is in a posture with its top facing upwards.

[0012] Optionally, based on the attitude category, the type of task to be performed is determined, including: When the posture category is the first posture category or the third posture category, it is determined that the grasping operation should be performed at this time; When the attitude category is the second attitude category, it is determined that the attitude adjustment operation should be performed.

[0013] Optionally, the robot end effector, controlled by the six degrees of freedom pose control, grasps the target fastener and performs a tightening operation, including: One or more candidate gripping poses for the target fastener are generated based on the six-degree-of-freedom pose. Collision detection is performed between each of the candidate grasping poses and the 3D point cloud data of the current working scene; When a collision-free candidate grasping pose exists, the robot end effector is controlled to grasp the target fastener according to the collision-free candidate grasping pose. When all candidate grasping poses have a collision risk, the grasping direction is resampled around the axis direction of the target fastener as a constraint to generate a new candidate grasping pose. The collision detection is then re-executed until a collision-free grasping pose is obtained, and then the robot end effector is controlled to grasp the target fastener. Move the target fastener to the target tightening position and perform the tightening operation.

[0014] Optionally, moving the target fastener to the target tightening position and performing a tightening operation includes: Based on the six-degree-of-freedom pose of the target fastener, determine the target tightening position and the direction of the tightening axis; The robot's end effector is controlled to move the grasped target fastener to the target tightening position and align the axis of the target fastener with the tightening axis. The robot's end effector drives the target fastener to feed and rotate along the tightening axis, performing a tightening action.

[0015] A second aspect of this application discloses a robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making, the system comprising: The point cloud acquisition module is used to acquire 3D point cloud data of the work scene; The point cloud segmentation module is used to segment the target point cloud data of the target fastener from the three-dimensional point cloud data; The pose estimation module is used to input the target point cloud data into a trained point cloud feature extraction network to regress and predict the spatial position of a predetermined key point on the target fastener; and to determine the six-degree-of-freedom pose of the target fastener based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener. An attitude determination module is used to determine the attitude category of the target fastener based on the six-degree-of-freedom pose. The attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping. The robot control module is used to determine the current task type to be performed based on the posture category, wherein the task type includes grasping task and posture adjustment task; when it is determined that the posture adjustment task should be performed, the robot end effector is controlled to perform the posture adjustment task on the target fastener, and after the posture adjustment task, the three-dimensional point cloud data of the target fastener is reacquired to update the six degrees of freedom pose and the posture category; when it is determined that the grasping task should be performed, the robot end effector is controlled to grasp the target fastener based on the six degrees of freedom pose and perform a tightening operation.

[0016] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making described in the first aspect of this application.

[0017] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making as described in the first aspect of this application.

[0018] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making as described in the first aspect of this application.

[0019] The embodiments of this application have the following advantages: This application employs a point cloud feature extraction network to regress predetermined key points, rather than directly performing global registration on the original point cloud. This avoids the dependence of traditional point cloud registration algorithms on complete, high-quality point clouds. Even if the point cloud contains holes, noise, or artifacts due to specular reflection on the surface of the metal fastener, the network can still stably predict the key point positions through the learned geometric structural features, and then calculate the six-degree-of-freedom pose, thereby achieving high-precision pose estimation without relying on high-cost hardware.

[0020] By performing semantic analysis on the six-degree-of-freedom pose to determine the pose category, the system automatically decides whether to perform a gripping operation or a pose adjustment operation based on the pose category. When the fastener's pose is not suitable for direct gripping, the system proactively performs a pose adjustment operation to change its spatial orientation. It can autonomously handle arbitrary placement postures such as lying on its side or flipping over, without manual intervention or production line downtime, thus improving the continuous operation capability of the automated production line. Simultaneously, after the pose adjustment operation, the system reacquires the 3D point cloud data of the target fastener to update the six-degree-of-freedom pose. This ensures that the pose information used in subsequent gripping operations reflects the actual state after pose adjustment, overcoming gripping failures caused by initial perception deviations or accumulated pose adjustment errors, and further improving the assembly success rate. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps of a robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making, provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The core concept of the robotic autonomous tightening method based on 3D point cloud keypoint detection and adaptive decision-making provided in this application is as follows: Keypoint regression is performed on the 3D point cloud of metal fasteners using a deep learning network. The six-degree-of-freedom pose is calculated using the geometric correspondence between keypoints and a standard model, thus avoiding the dependence of traditional point cloud registration algorithms on the complete point cloud. Based on this, semantic understanding of the pose information is performed to determine the spatial pose category of the fastener. Then, based on the pose category, a finite state machine is used to autonomously decide whether to perform grasping or pose adjustment operations, forming a closed-loop control framework of "perception-decision-execution-re-perception". This technical concept combines deep learning-driven visual perception with semantic decision-based adaptive control, aiming to solve the technical problems of difficult pose detection and inability to autonomously handle non-standard poses of reflective metal fasteners in bulk placement scenarios, achieving high-precision and robust autonomous assembly under low-cost hardware conditions. Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making, as provided in an embodiment of this application. Figure 1 As shown, the method may include steps S110 to S180: Step S110: Obtain the 3D point cloud data of the work scene.

[0025] In this step, the work scene refers to the working area where the robot performs the fastener assembly task. This area may include the target fastener, fixtures for placing or constraining the fastener, a work surface, and surrounding environmental objects. Three-dimensional point cloud data refers to the collection of points obtained after scanning the work scene using a three-dimensional vision sensor. Each point contains its coordinate information in three-dimensional space, and optionally, it may also contain color or intensity information. The three-dimensional vision sensor can be a structured light camera, a binocular camera, or a LiDAR, etc., capable of outputting three-dimensional point cloud data. This embodiment does not limit the specific type of sensor used.

[0026] In some implementations, after acquiring the 3D point cloud data, the raw point cloud data can be preprocessed, including but not limited to coordinate transformation (converting the point cloud in the sensor coordinate system to the robot base coordinate system or the world coordinate system), timestamp alignment, etc., to facilitate subsequent processing steps.

[0027] Step S120: Segment the target point cloud data of the target fastener from the three-dimensional point cloud data.

[0028] In this step, the point cloud region corresponding to the target fastener is extracted from the 3D point cloud data of the work scene containing various objects, thereby eliminating interference from background point clouds such as fixtures, work surfaces, and other irrelevant parts. The target fastener refers to the single bolt or nut to be operated. Segmentation can be performed using spatial location-based methods or point cloud feature clustering methods; this embodiment does not limit the specific method used.

[0029] Step S130: Input the target point cloud data into a trained point cloud feature extraction network to regress and predict the spatial location of predetermined key points on the target fastener.

[0030] In this step, the point cloud feature extraction network refers to a deep learning-based neural network model capable of extracting high-level semantic features from 3D point clouds. This network takes point cloud data as input and directly outputs the predicted 3D coordinates of predetermined key points on the fastener.

[0031] Among them, predetermined key points refer to several feature locations on a fastener with clear geometric semantics, such as the center point of the top of the bolt head, the center point of the end of the shank, and the middle position point of the shank; or the center point of the top surface, the center point of the bottom surface, and the middle position point of the side surface of the nut. The point cloud feature extraction network learns from a large amount of sample data labeled with the locations of these key points, and can directly regress the spatial locations of these key points from the input point cloud data.

[0032] Understandably, unlike traditional point cloud registration methods, this embodiment uses key point regression. Even when the point cloud contains holes, noise, or artifacts due to specular reflection from the metal surface, the network can still stably predict the key point positions based on the learned overall geometric features of the fastener.

[0033] Step S140: Determine the six-degree-of-freedom pose of the target fastener based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener.

[0034] In this step, the standard model refers to the three-dimensional geometric model of the target fastener in a standard orientation. This model has predefined key point positions that correspond one-to-one with predetermined key points. The standard model can be generated based on the fastener's CAD model or obtained by scanning the fastener in a standard orientation.

[0035] The transformation relationship refers to the rotation and translation transformation required to map the spatial position of a predetermined key point to the position of the corresponding key point on a standard model. Since the spatial position of the predetermined key point is described in the coordinate system of the 3D point cloud data, while the position of the corresponding key point on the standard model is described in the model's own coordinate system, the transformation relationship between the two reflects the actual pose of the target fastener in the operational scene. This transformation relationship can be solved using the Singular Value Decomposition (SVD) algorithm.

[0036] Step S150: Determine the attitude category of the target fastener based on the six-degree-of-freedom pose. The attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping.

[0037] In this step, the attitude category refers to the semantic classification result of the current spatial orientation of the target fastener, which is used to characterize whether the fastener is in a spatial attitude that the robot's end effector can directly and stably grasp. For example, for a bolt, if its axis is roughly vertical and the head is facing upwards, the robot arm can directly grip the head from above, and this attitude is suitable for direct grasping; if the bolt is horizontal and lying on its side, direct grasping is more difficult or cannot be stably grasped, and this attitude is not suitable for direct grasping.

[0038] Posture categories can include various types, such as "top up," "top down," and "sideways," or they can be simplified to "gripable" and "non-gripable." The posture category can be determined based on the relative relationship between the fastener's central axis and the direction of gravity.

[0039] Step S160: Determine the current task type to be performed based on the posture category, wherein the task type includes grasping task and posture adjustment task.

[0040] In this step, the job type refers to the type of operation that the robot should perform next, determined based on the current posture category of the target fastener. A grasping operation refers to the operation of directly grasping the target fastener; a posture adjustment operation refers to the operation of changing the current spatial orientation of the target fastener, for example, by touching or moving the fastener with the robot's end effector to make it roll or rotate, thereby adjusting it to a suitable posture for grasping.

[0041] The decision is made based on the attitude category. If the attitude category indicates that the fastener is currently in an attitude suitable for direct gripping, then the gripping operation is executed; if the attitude category indicates that the fastener is currently in an attitude unsuitable for direct gripping, then the attitude adjustment operation is executed. This decision-making process can be implemented using a finite state machine (FSM), where different attitude categories correspond to different state transition paths.

[0042] Step S170: When it is determined that the attitude adjustment operation should be performed, control the robot end effector to perform the attitude adjustment operation on the target fastener, and after the attitude adjustment operation, reacquire the three-dimensional point cloud data of the target fastener to update the six-degree-of-freedom pose and the attitude category.

[0043] In this step, the robot's end effector is controlled to perform a touching or prying operation on the target fastener to change its spatial orientation. The specific operation method of the attitude adjustment operation depends on the current attitude category of the fastener and the physical constraints of the working environment. For example, when the nut is in a sideways posture, the robot can control the end effector to touch the side edge of the nut with a specific vector, causing it to roll to an upright posture.

[0044] After performing the attitude adjustment operation, the process returns to step S110 to reacquire the 3D point cloud data of the work scene, and then sequentially executes steps S120 to S150 to update the six-DOF pose and attitude category of the target fastener, thereby confirming whether the attitude adjustment has achieved the expected effect. If the updated attitude category is still a non-standard attitude, the attitude adjustment operation can be repeated until the attitude category changes to a pose suitable for direct grasping. Through the above-described closed-loop control of "perception-adjustment-re-perception", this application can overcome the positioning deviation that may be caused by a one-time perception and the pose uncertainty that may be caused by the attitude adjustment operation itself.

[0045] Step S180: When it is determined that the gripping operation should be performed, the robot end effector is controlled according to the six degrees of freedom pose to grip the target fastener and perform a tightening operation.

[0046] In this step, a gripping path and gripping posture are generated based on the six-degree-of-freedom pose. The robot's end effector is then controlled to move to the location of the target fastener and grip it with an appropriate clamping force. After gripping, the fastener is moved to the target tightening position, where it is automatically aligned with the threaded hole and rotated for tightening.

[0047] The technical solution adopted in this embodiment uses keypoint regression instead of traditional point cloud registration to determine the six-degree-of-freedom pose. Even when point cloud quality deteriorates due to metal reflection, it maintains stable pose estimation accuracy without relying on high-cost hardware such as six-dimensional force sensors. Simultaneously, it automatically makes decisions and performs attitude adjustments based on attitude category, autonomously handling arbitrary orientations such as side-lying and flipping, avoiding production line downtime caused by non-standard attitudes and improving the continuous operation capability of automated production lines. Furthermore, after attitude adjustment, the three-dimensional point cloud data of the target fastener is reacquired to update the six-degree-of-freedom pose, overcoming the problem of grasping failure caused by initial perception deviations or accumulated attitude adjustment errors, further improving the assembly success rate.

[0048] In an optional embodiment, step S120 above, "segmenting the target point cloud data of the target fastener from the three-dimensional point cloud data," may include steps S120-1 to S120-3: Step S120-1: Perform spatial filtering on the three-dimensional point cloud data according to the preset spatial range parameters to filter out the background point cloud outside the area where the target fastener is located.

[0049] In this step, the original 3D point cloud data is cropped using preset spatial range parameters, retaining only the point cloud within the target area, thereby excluding background objects irrelevant to the task in the work scene. The preset spatial range parameters define the 3D spatial region where the target fastener may appear in the work scene. This region is typically related to the robot's workspace, the fixture's placement, and the field of view of the 3D vision sensor. For example, a cuboid region of interest can be defined, covering the fixture's location and a certain surrounding area. Only the point cloud data within this region is retained, while point clouds outside the region (such as distant workbench edges, other equipment, etc.) are filtered out.

[0050] Spatial filtering can reduce the amount of data that needs to be processed in subsequent steps, while also preventing irrelevant point clouds from interfering with subsequent segmentation and pose estimation.

[0051] Step S120-2: Remove outliers from the filtered point cloud data.

[0052] In this step, outliers refer to points in the spatially filtered point cloud that deviate from the main body of the point cloud and are isolated in space. Outliers may be caused by measurement noise from the 3D sensor at the edges of reflective metallic surfaces, misdetection of suspended particles in the environment as point clouds, or shot noise from the sensor itself. These outliers can interfere with subsequent clustering segmentation and keypoint regression, and therefore need to be removed.

[0053] Further, step S120-2 specifically includes: projecting simulated light rays from a preset viewpoint position to each point in the filtered point cloud data; if there are other point cloud occlusions on the path connecting the current point and the preset viewpoint position, then retaining the target point; if there are no other point cloud occlusions on the path connecting the current point and the preset viewpoint position, then identifying the target point as an outlier and removing it.

[0054] The preset viewpoint position refers to the spatial position of the 3D vision sensor when acquiring point clouds, and this position can be obtained through the sensor's calibration parameters. A simulated ray is a virtual ray originating from the viewpoint position and passing through a point in the point cloud. The existence of other point cloud occlusions on the path connecting the current point and the preset viewpoint position means that if light emitted from the viewpoint illuminates the fastener surface and is reflected back to the sensor at that point, and if that point truly belongs to the fastener surface, then there should be continuous surface point cloud occlusions on the path from the viewpoint to that point. The absence of other point cloud occlusions on the path connecting the current point and the preset viewpoint position means that if there are no other point cloud occlusions on the path from the viewpoint to a point, then that point is actually an isolated point suspended in space, does not constitute a continuous object surface, and is an invalid measurement point.

[0055] Thus, this outlier removal method utilizes the imaging geometry principle of 3D point clouds, which can effectively identify and remove spatially isolated points caused by sensor noise, without mistakenly deleting valid points located on the surface of the object.

[0056] Step S120-3: Cluster and segment the point cloud data after filtering out outliers based on the point cloud density distribution to obtain the target point cloud of the target fastener.

[0057] In this step, based on the spatial distribution density of points in the point cloud, densely clustered points are divided into the same cluster, thereby separating the point clouds corresponding to different objects. Specifically, a density-based spatial clustering algorithm can be used to segment the point cloud into instances. This algorithm can automatically identify clusters formed by dense regions based on the local density distribution of the point cloud, and regard regions with low density as background or noise.

[0058] For scenarios involving multiple fasteners or where fasteners coexist with fixtures, clustering segmentation can divide each independent object instance into separate point cloud clusters, thereby extracting the target point cloud of a single target fastener. The neighborhood radius threshold of the density-based spatial clustering algorithm can be adaptively set according to the point cloud density distribution, or a fixed value can be preset according to the actual size of the target fastener; this embodiment does not impose such limitations.

[0059] In an optional implementation, after step S120-3, a step of voxel downsampling the point cloud may be included, that is, using a voxel grid to downsample the point cloud to reduce the point cloud density and the amount of data, thereby improving the computational efficiency of subsequent network inference.

[0060] The technical solution adopted in this embodiment quickly eliminates background point clouds irrelevant to the task through spatial filtering, reducing the computational burden of subsequent processing. Outlier removal based on ray projection accurately identifies and removes isolated invalid points caused by sensor noise using the geometric principles of 3D imaging, preserving effective measurement information of the object surface and avoiding interference from outliers in subsequent pose estimation. This provides high-quality point cloud input for stable inference of the keypoint regression network. Density-based clustering segmentation accurately extracts individual target fasteners, allowing subsequent processing to focus on the point cloud of a single part, avoiding pose estimation ambiguities caused by multiple objects. The above segmentation process together constitutes a complete processing link from the original point cloud to the point cloud of a single part, ensuring the reliability of the perception stage in complex operational scenarios.

[0061] In an optional embodiment, the point cloud feature extraction network is trained according to steps A1 to A6: Step A1: Construct a training dataset. Each training sample in the training dataset includes the 3D point cloud data of the sample fastener and the corresponding annotation information. The annotation information includes the true coordinates of multiple predetermined key points on the sample fastener and the true height of the sample fastener.

[0062] In this step, the training dataset is a collection of samples used to train the point cloud feature extraction network. Each training sample comes from 3D point cloud data obtained by 3D scanning of sample fasteners, along with corresponding ground truth information annotated manually or semi-automatically.

[0063] Specifically, the sample fasteners can be bolts or nuts of the same type as the target fasteners, such as M6 socket head cap screws. The 3D point cloud data of the sample fasteners can be acquired using a 3D vision sensor under different orientations and lighting conditions to cover various placement states that may occur in actual operations. To improve the network's generalization ability, the training data may also include samples generated after data augmentation of the original point cloud. This data augmentation includes, but is not limited to, random rotation, random scaling, random translation, and adding simulated noise.

[0064] Multiple predefined key points in the annotation information correspond one-to-one with the predefined key points. For example, for bolt-type fasteners, predefined key points may include the center point of the top of the head, the center point of the end of the shank, and the middle position point of the shank, corresponding to the top feature point, bottom feature point, and middle feature point of the standard model, respectively. Annotators need to accurately mark the three-dimensional coordinates of these key points in the point cloud. The actual height refers to the physical dimension of the sample fastener along its principal axis, such as the total length of the bolt or the thickness of the nut, which can be obtained using measuring tools such as calipers or directly read from the CAD model. Height information, as one of the supervision signals, is used to constrain the network's ability to perceive the overall scale of the fastener.

[0065] Step A2: Input the 3D point cloud data of the sample fastener into the point cloud feature extraction network to be trained to obtain the predicted coordinates of the multiple predetermined key points and the predicted height of the sample fastener.

[0066] In this step, the point cloud feature extraction network to be trained refers to the initial network model whose parameters have not yet been optimized to their optimal state. The point cloud data of the training samples is input into the network. After forward propagation, the network outputs the predicted coordinates of multiple predetermined key points, which are the network's estimates of the three-dimensional position of each key point; and the predicted height of the sample fastener, which is the network's estimate of the fastener's dimension in the spindle direction.

[0067] The architecture of the point cloud feature extraction network can be an encoder-decoder structure based on PointNet++. The encoder is used to extract local and global geometric features of the point cloud layer by layer, and the decoder is used to regress the key point coordinates and height values ​​from the features.

[0068] Step A3: Calculate the key point loss based on the difference between the predicted coordinates and the actual coordinates of the multiple predetermined key points.

[0069] In this step, the keypoint loss is used to measure the deviation between the network's predicted keypoint locations and the actual labeled locations. The core function of this loss function is to constrain the network's output keypoint coordinates to be as close as possible to the true values ​​in spatial location.

[0070] Specifically, for each predetermined keypoint, the Euclidean or Manhattan distance between its predicted and true coordinates is calculated. Then, the distances of all keypoints are averaged to obtain the keypoint loss. For example, the L1 norm (i.e., the sum of absolute differences along each coordinate axis) can be used to measure the distance between the predicted and true coordinates of a single keypoint, and then the average distances of all keypoints are calculated. The smaller the keypoint loss, the more accurate the network's prediction of keypoint locations. By minimizing this loss, the network is driven to learn how to accurately locate geometrically semantic features on fasteners from the point cloud.

[0071] Step A4: Calculate the direction loss based on the spatial collinearity between the predicted coordinates of the multiple predetermined key points.

[0072] In this step, the orientation loss is used to measure whether the network's predicted keypoints satisfy the preset spatial geometric constraints. For fasteners, their different feature points along the main axis (such as the top center point, middle point, and bottom center point) are approximately collinear in space. This loss function utilizes this structural prior to enhance the network's ability to perceive the fastener's orientation.

[0073] Specifically, the orientation loss can be calculated as follows: First, select two pairs of keypoints from the predicted keypoints and calculate the unit orientation vector between each pair. Then, calculate the dot product between these two unit orientation vectors. The orientation loss is equal to 1 minus this dot product. When the two unit orientation vectors are aligned, the dot product approaches 1, and the orientation loss approaches 0, indicating a high degree of collinearity among the three keypoints. When the two unit orientation vectors are not aligned, the dot product is less than 1, and the orientation loss is greater than 0, indicating a low degree of collinearity among the three keypoints. By minimizing the orientation loss, the network is driven to learn and predict the positions of keypoints that satisfy the collinear geometric constraints, thereby making the prediction results more consistent with the actual physical structure of the fastener and improving the geometric rationality of the keypoint prediction.

[0074] Step A5: Calculate the size loss based on the difference between the predicted height and the actual height.

[0075] In this step, the size loss is used to measure the degree of deviation between the fastener height predicted by the network and the actual height. The role of this loss function is to constrain the network's perception of the overall size of the fastener.

[0076] Specifically, the dimensional loss can be calculated as the absolute value of the difference between the predicted height and the actual height, or it can be other forms of distance metric such as squared error. The height of a fastener is one of its most fundamental physical dimensional parameters, remaining constant within fasteners of the same specification. By introducing the dimensional loss, the network is driven to learn to predict height values ​​consistent with the actual physical dimensions, thus providing additional scale constraints beyond keypoint prediction and enhancing the network's ability to model the overall geometric features of the fastener.

[0077] Step A6: Update the network parameters of the point cloud feature extraction network to be trained based on the key point loss, the orientation loss, and the size loss.

[0078] In this step, the network parameters are updated based on the backpropagation algorithm and the gradient descent optimization method. Specifically, the keypoint loss, orientation loss, and size loss are weighted and summed to obtain the total loss function; then, the gradient of the total loss function with respect to the parameters of each layer of the network is calculated; finally, the network parameters are updated along the gradient descent direction, so that the value of the total loss function gradually decreases. The above process is performed iteratively on the training dataset until the network converges or reaches the preset number of training epochs.

[0079] In one alternative implementation, the keypoint loss, orientation loss, and size loss can be weighted and combined using different weighting coefficients. For example, the keypoint loss, as the primary monitoring signal, can be assigned a higher weight; while the orientation loss and size loss, as auxiliary constraints, can be assigned relatively lower weights. The specific values ​​of the weighting coefficients can be determined through experimental debugging, and this application does not limit them.

[0080] In one alternative implementation, the input channels of the point cloud feature extraction network can be configured according to the type of fastener. For fasteners with color markings on the surface (such as bolts with color-coded heads to distinguish specifications), the network input can be a multidimensional tensor containing three-dimensional coordinates and color channels (e.g., a [N, 6]-dimensional tensor containing x, y, z coordinates and R, G, B color channels). The network can use color information to help distinguish different parts of the fastener (e.g., the head and the shank). For fasteners that are purely metallic and have no color markings, the network input can be a tensor containing only three-dimensional coordinates (e.g., a [N, 3]-dimensional tensor). The network predicts key points purely based on the geometric structure.

[0081] The technical solution employed in this embodiment provides ample supervision information to the network by constructing a training dataset containing dual annotations of keypoint coordinates and heights. Through joint optimization of three supervision signals—keypoint loss, orientation loss, and size loss—the network learns not only the accurate location of individual keypoints but also the spatial collinearity between keypoints and the overall scale information of the fastener. The synergistic effect of the three losses enables the trained point cloud feature extraction network to stably predict the keypoint locations that conform to the fastener's geometry, even when the point cloud contains noise or partial missing data, providing accurate input for subsequent six-degree-of-freedom pose calculation.

[0082] In one optional embodiment, the predetermined key points include a first key point, a second key point, and a third key point, wherein the first key point, the second key point, and the third key point correspond to the top feature position, the middle feature position, and the bottom feature position of the target fastener, respectively.

[0083] The step S140 above, "determining the six-degree-of-freedom pose of the target fastener based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener," may include steps S140-1 to S140-5: Step S140-1: Determine the feature point type corresponding to each of the predetermined key points according to the standard model. The feature point type includes top feature point, middle feature point and bottom feature point.

[0084] In this step, the standard model is pre-labeled with feature point locations corresponding one-to-one with the keypoints predicted by the point cloud feature extraction network, and each feature point is assigned a semantic type label. For example, for bolt-type fasteners, the top feature point corresponds to the center point of the bolt head end face, the bottom feature point corresponds to the center point of the bolt shank end face, and the middle feature point corresponds to the point at the axial middle position of the bolt shank. For nut-type fasteners, the top feature point corresponds to the center point of the nut top surface, the bottom feature point corresponds to the center point of the nut bottom surface, and the middle feature point corresponds to the point at the middle height position of the nut side surface. Through the definition of feature point types in the standard model, the three spatial points predicted by the network are given semantic meanings of "top," "middle," and "bottom," thus providing a clear geometric reference for subsequent coordinate system construction.

[0085] Step S140-2: Determine the positions of the first key point, the second key point, and the third key point from the spatial positions of the predetermined key points according to the feature point type.

[0086] In step S130, the point cloud feature extraction network outputs the spatial locations of three predetermined key points, but the semantics of these three points are not yet clear at this time. Based on the feature point types defined in the standard model in step S140-1, the three spatial points output by the network can be determined as the first key point (corresponding to the top feature point), the second key point (corresponding to the middle feature point), and the third key point (corresponding to the bottom feature point) according to their relationship with the top, middle, and bottom feature points on the standard model in the point cloud.

[0087] Step S140-3: Determine a first direction vector based on the position of the first key point and the position of the second key point, and determine a second direction vector based on the position of the second key point and the position of the third key point.

[0088] In this step, a direction vector refers to a vector pointing from one key point in space to another. Specifically, the first direction vector points from the top key point to the middle key point, i.e., from the position of the first key point to the position of the second key point; the second direction vector points from the middle key point to the bottom key point, i.e., from the position of the second key point to the position of the third key point. Both direction vectors extend approximately along the principal axis of the fastener.

[0089] Step S140-4: Determine the axial direction of the target fastener based on the average direction of the first direction vector and the second direction vector, and determine the spatial position of the key point corresponding to the top feature point of the standard model among the predetermined key points as the origin of the local coordinate system.

[0090] In this step, to obtain a more robust axis orientation estimate, the average direction of the first and second direction vectors is taken as the axis orientation of the fastener. Specifically, the first and second direction vectors are first normalized, then the average of the two unit vectors is calculated, and this average is normalized again. The resulting unit vector is the axis orientation vector. This axis orientation vector points from the bottom to the top. By taking the average, the error introduced by the prediction deviation of a single direction vector can be effectively suppressed.

[0091] The origin of the local coordinate system is defined as the spatial location of the top keypoint, that is, the actual predicted location of the keypoint corresponding to the top feature point in the standard model. The origin is defined at the top rather than the geometric center or other locations because the top is usually the part that the robot directly contacts during grasping, which facilitates the generation of subsequent grasping poses.

[0092] Step S140-5: Construct a local coordinate system for the target fastener based on the axial direction and the origin, and determine the six-degree-of-freedom pose based on the rotation and displacement relationships of the local coordinate system relative to the reference coordinate system; wherein, the reference coordinate system is the coordinate system in which the three-dimensional point cloud data is located.

[0093] In this step, after determining the axis direction and the origin position, the local coordinate system of the target fastener can be constructed. The axis direction serves as the Z-axis direction of the local coordinate system, and the origin position serves as the origin position of the local coordinate system. The X and Y axes of the local coordinate system can be determined through orthogonalization methods, such as using Singular Value Decomposition (SVD) or Schmidt orthogonalization to construct an orthonormal basis from the axis direction and any initial vector, thus obtaining the complete local coordinate system.

[0094] The six-DOF pose includes three translational degrees of freedom (i.e., the position of the origin in the reference coordinate system, given by the spatial position of the top keypoint) and three rotational degrees of freedom (i.e., the orientation of the local coordinate system relative to the reference coordinate system, determined by the axial direction and the X / Y axis direction). This six-DOF pose fully describes the spatial position and orientation of the target fastener in the working scene, providing input for the robot's grasping path planning and end effector pose control.

[0095] The technical solution adopted in this embodiment uses a standard model to assign semantic types of top, middle, and bottom to the three key points predicted by the network, giving the disordered output points clear geometric and physical meanings. By averaging the two direction vectors to determine the axis direction, the influence of single prediction error on axis estimation is suppressed, improving the stability of pose calculation. The origin is defined at the top key point, directly serving the subsequent generation of grasping pose without additional conversion. A local coordinate system is constructed using the axis direction and the origin, which fully describes the six degrees of freedom information of the fastener in three-dimensional space, providing accurate pose input for the robot's accurate grasping and assembly operations.

[0096] In an optional embodiment, step S150 above, "determining the attitude category of the target fastener based on the six-degree-of-freedom pose," may include steps S150-1 to S150-5: Step S150-1: Extract the central axis direction vector of the target fastener from the six-degree-of-freedom pose. The central axis direction vector points from the bottom end to the top end of the target fastener.

[0097] In this step, the six-degree-of-freedom pose includes the position and orientation information of the target fastener in the reference coordinate system. The central axis direction vector is a unit vector pointing from the bottom to the top along the principal axis of the target fastener, and its direction is consistent with the axis direction determined in step S140. The central axis direction vector can be extracted from the rotation matrix of the six-degree-of-freedom pose. This vector serves as the basis for determining the spatial orientation of the fastener, and subsequent steps will use the relative relationship between this vector and the gravity direction vector to determine the attitude category.

[0098] Step S150-2: Obtain the gravity direction vector and calculate the angle between the central axis direction vector and the gravity direction vector.

[0099] In this step, the gravity direction vector refers to the direction of gravitational acceleration in the world coordinate system or the robot's base coordinate system, typically a vertically downward unit vector. This vector can be obtained through the robot's attitude sensors and inertial measurement unit, or it can be determined through prior calibration information of the work scenario.

[0100] The angle between the central axis direction vector and the gravity direction vector can be calculated by the dot product of the two unit vectors. This angle reflects the degree of inclination of the fastener's main axis relative to the vertical direction. When the fastener's main axis is in the same direction as gravity, the angle is 0°; when the fastener's main axis is opposite to gravity, the angle is 180°; and when the fastener's main axis is horizontal, the angle is 90°.

[0101] Step S150-3: When the included angle is less than or equal to the first preset angle threshold, the posture category is determined to be the first posture category, which indicates that the target fastener is in a posture with the top facing down.

[0102] In this step, the first preset angle threshold is used to define the upper limit of the angle at which the fastener is in a top-down orientation. When the angle between the central axis direction vector and the gravity direction vector is small (i.e., the fastener's main axis is nearly vertical and basically consistent with the gravity direction), it indicates that the fastener's top is down and its bottom is up. In this orientation, the fastener is in an upright state, and the robot's end effector can stably grasp it from above or the side. The specific value of the first preset angle threshold can be set according to the actual working scenario and the type of fastener.

[0103] Step S150-4: When the included angle is greater than the first preset angle threshold and less than the second preset angle threshold, the posture category is determined to be the second posture category; the second posture category indicates that the target fastener is in a side-lying posture, and the first preset angle threshold is less than the second preset angle threshold.

[0104] In this step, when the angle between the central axis direction vector and the gravity direction vector is between the first preset angle threshold and the second preset angle threshold, it indicates that the fastener's main axis is neither close to vertical nor opposite to the gravity direction, but is roughly horizontal or tilted, meaning the fastener is lying on its side on the worktable. In this posture, the fastener's axis is roughly parallel to the tabletop, and its top and bottom ends are at approximately the same height, making it difficult for the robot to hold it stably from above. The specific value of the second preset angle threshold can be set according to actual needs. The first and second preset angle thresholds together define the angle range of the side-lying posture.

[0105] Step S150-5: When the included angle is greater than or equal to the second preset angle threshold, the posture category is determined to be the third posture category, which indicates that the target fastener is in a posture with the top facing upwards.

[0106] In this step, when the angle between the central axis direction vector and the gravity direction vector is large (i.e., the fastener's main axis is nearly vertical and roughly opposite to the gravity direction), it indicates that the fastener's top is facing upwards and its bottom is facing downwards. In this posture, the fastener is upright, and the robot's end effector can directly grip the head of the fastener from above, making it an easy-to-grasp posture. The third posture category corresponds to the first posture category. The first posture category is top-down (main axis direction is consistent with the gravity direction), and the third posture category is top-up (main axis direction is opposite to the gravity direction). The angle ranges between the two are approximately 0 degrees and 180 degrees, respectively.

[0107] In one optional implementation, the first preset angle threshold can be set to 60°, and the second preset angle threshold can be set to 120°. In this case, the angle between the central axis and the direction of gravity is determined to be a first posture category (top down) if it is between 0° and 60° (inclusive), a second posture category (side-lying) if it is between 60° and 12°, and a third posture category (top up) if it is between 120° and 180° (inclusive).

[0108] The technical solution of this embodiment maps the orientation information in the six-degree-of-freedom pose to the angle between the fastener's central axis and the direction of gravity, and discretizes and classifies continuous angle values ​​using a preset angle threshold, quantifying the fastener's spatial orientation into a finite number of attitude categories with clear semantics. This classification method is simple and effective, requiring no complex geometric reasoning or additional sensor data. It can complete the attitude determination using only the existing six-degree-of-freedom pose information and the known direction of gravity, providing semantic input for subsequent job type decisions.

[0109] In an optional embodiment, step S160 above, "determining the type of task to be performed based on the attitude category," may include steps S160-1 to S160-2: Step S160-1: When the posture category is the first posture category or the third posture category, determine that the grasping operation should be performed.

[0110] The first posture category indicates that the target fastener is in a downward-facing position, while the third posture category indicates that the target fastener is in a downward-facing position. When the target fastener is in either of these posture categories, the angle between its main axis and the vertical direction is small, meaning the fastener is generally upright. In this upright posture, the top and bottom of the target fastener are at different heights, allowing the robot's end effector to easily approach and stably clamp the fastener from above or from the side.

[0111] Specifically, for the third posture category (top up), the fastener head faces upward and the rod faces downward. The robot's end effector can directly grip the fastener head from directly above, which is the ideal gripping posture. For the first posture category (top down), the fastener head faces downward and the rod faces upward. The robot's end effector may need to grip the fastener rod from the side or above, but stable gripping is still possible. Therefore, when the posture category is the first or third posture category, no posture adjustment is required, and the gripping process can proceed directly.

[0112] It should be noted that, for the first posture category and the third posture category, different grasping strategies (such as different gripping positions or gripping directions) can be selected according to the differences in the specific posture. This strategy can be implemented through preset grasping pose generation logic, which is not limited in this application.

[0113] Step S160-2: When the attitude category is the second attitude category, determine that the attitude adjustment operation should be performed.

[0114] The second posture category indicates that the target fastener is in a side-lying posture, meaning the angle between the fastener's spindle direction and the vertical direction is moderate, and the fastener lies roughly horizontally on the worktable or fixture surface. In this posture, the top and bottom ends of the fastener are at approximately the same height, and the entire fastener is placed horizontally.

[0115] When a fastener is in a sideways position, the robot's end effector struggles to perform stable gripping directly. Firstly, because the fastener is in contact with the worktable or fixture surface, the end effector cannot easily reach in from below or the side for clamping. Secondly, even if clamping is possible, the gripping stability is poor because the gripping point may deviate from the fastener's center of gravity or axis, making slippage or overturning during gripping easy. Therefore, when the posture category is the second posture category, direct gripping is deemed unsuitable. Instead, posture adjustment should be prioritized. This involves using the robot's end effector to touch or manipulate the fastener, causing it to flip from a sideways posture to an upright posture (i.e., changing to the first or third posture category). Then, sensing and pose updates are performed again to create conditions for subsequent gripping operations.

[0116] The technical solution of this embodiment maps three posture categories to two job types. The first posture category (top down) and the third posture category (top up) directly trigger the grasping job, while the second posture category (sideways) triggers the posture adjustment job. This establishes a clear correspondence between posture semantics and job decision-making, enabling the robot to autonomously select a reasonable job path based on the actual spatial posture of the target fastener. For fasteners already in a graspable upright posture, grasping is performed directly, improving job efficiency. For fasteners in a sideways posture, posture adjustment is performed first to flip them into an upright posture before grasping, ensuring a high grasping success rate and the reliability of subsequent assembly operations. The above decision-making process requires no manual judgment or intervention, achieving fully autonomous job flow control.

[0117] In an optional embodiment, step S180 above, "controlling the robot's end effector to grasp the target fastener according to the six-degree-of-freedom pose and performing a tightening operation," may include steps S180-1 to S180-5: Step S180-1: Generate one or more candidate gripping poses for the target fastener based on the six-degree-of-freedom pose.

[0118] In this step, the grasping pose refers to the spatial position and orientation that the robot's end effector should reach when grasping the target fastener, which is also described by six degrees of freedom parameters. Based on the six-degree-of-freedom pose of the target fastener determined in step S140, one or more selectable grasping poses can be calculated.

[0119] Specifically, the generation of the gripping pose needs to consider the following factors: the spatial position and orientation of the target fastener (provided by the six-DOF pose), the geometry and gripping method of the end effector, and the preset gripping strategy (e.g., gripping the head from directly above, gripping the rod from the side, etc.). For the same target fastener, there may be multiple feasible gripping directions. For example, it can be gripped from top to bottom along the axis, or it can be gripped from the side along a direction perpendicular to the axis. Multiple candidate gripping poses can be generated according to the preset gripping strategy, and the multiple candidate gripping poses can be arranged in a preset priority order, such as prioritizing gripping poses along the axis, with a large gripping area, and high stability.

[0120] Step S180-2: Perform collision detection between each candidate grasping pose and the 3D point cloud data of the current working scene.

[0121] In this step, collision detection is used to verify whether the candidate grasping pose will cause physical interference between the robot's end effector and obstacles in the work environment (such as jigs, worktables, other parts, etc.).

[0122] Specifically, the 3D model of the robot's end effector can be placed at the position and orientation specified by the candidate grasping pose, and then the model can be spatially intersected with the 3D point cloud data of the current working scene. If the end effector model overlaps with any point cloud in space or the distance is less than the preset safety gap, the candidate grasping pose is determined to have a collision risk; otherwise, it is determined to be a collision-free candidate grasping pose.

[0123] Step S180-3: When there is a collision-free candidate grasping pose, control the robot end effector to grasp the target fastener according to the collision-free candidate grasping pose.

[0124] In this step, if there are one or more candidate grasping poses that pass collision detection, the optimal grasping pose is selected (for example, the one with the highest priority among multiple collision-free poses), and the robot end effector is controlled to move to the grasping pose to perform the grasping operation, that is, to close the gripper of the end effector to hold the target fastener.

[0125] Step S180-4: When all candidate grasping poses have a collision risk, the grasping direction is resampled around the axis direction of the target fastener as a constraint to generate a new candidate grasping pose, and the collision detection is re-executed until a collision-free grasping pose is obtained, and then the robot end effector is controlled to grasp the target fastener.

[0126] In this step, if all candidate grasping poses fail the collision detection, it indicates that the preset grasping strategy cannot be directly executed in the current scenario. At this point, the resampling obstacle avoidance process begins. Specifically, while maintaining the six degrees of freedom pose of the target fastener, rotational sampling is performed along its axial direction, generating new grasping pose candidates at different circumferential angles. Since the most important degree of freedom in the grasping pose is the direction in which the end effector approaches the target (i.e., the grasping direction), and this direction is usually perpendicular (lateral clamping) or parallel (axial clamping) to the axial direction of the target fastener, rotational sampling can be performed on the grasping direction within a plane perpendicular to the axis or within a plane containing the axis, generating new grasping directions not included in the original candidate set. For example, for fasteners requiring lateral clamping, a grasping pose candidate can be generated at regular angle intervals (e.g., 30 degrees or 45 degrees) within a plane perpendicular to the axis, thus searching within the complete 360-degree circumferential range.

[0127] After generating new candidate grasping poses, collision detection in step S180-2 is re-executed, and the above process is repeated until a collision-free grasping pose is found. If a collision-free pose cannot be found after a preset number of resampling operations, it can be determined that safe grasping is not possible in the current work scenario, and an abnormal alarm is triggered or manual intervention is required. Through the above resampling obstacle avoidance mechanism, obstacles in the work scenario can be effectively avoided while ensuring the accuracy of the grasping pose, thus achieving safe grasping.

[0128] Step S180-5: Move the target fastener to the target tightening position and perform the tightening operation.

[0129] In this step, after successfully gripping the target fastener, the robot's end effector moves the fastener from the gripping position to the target tightening position. The target tightening position refers to the spatial location of the threaded hole or nut to be assembled, where the target fastener needs to be threadedly connected to the mating part. Upon reaching the target tightening position, the end effector rotates the fastener to achieve thread engagement and tightening.

[0130] In an optional embodiment, step S180-5 may specifically include steps S180-5-1 to S180-5-3: Step S180-5-1: Determine the target tightening position and tightening axis direction based on the six-degree-of-freedom pose of the target fastener.

[0131] In this step, the target tightening position can be determined based on the spatial position of the mating parts (such as the threaded hole to be tightened or the nut to be mated) in the work scenario. This position can be obtained in advance through visual perception or through prior calibration information of the work scenario. The tightening axis direction refers to the direction of fastener feed during threaded connection, which is usually consistent with the axis direction of the threaded hole of the mating part, for example, vertically downward in a vertical installation scenario. The target tightening position and the tightening axis direction together constitute the target pose of the tightening operation, and the robot end effector needs to adjust the gripped fastener to this target pose.

[0132] Step S180-5-2: Control the robot end effector to move the grasped target fastener to the target tightening position and align the axis of the target fastener with the tightening axis.

[0133] In this step, the robot plans a motion path from the current gripping position to the target tightening position based on the target tightening position and the tightening axis direction. It controls the end effector to move the fastener to the target tightening position and adjusts the end effector's attitude to align the axis direction of the gripped fastener with the tightening axis direction. This alignment operation ensures that subsequent tightening actions are performed along the correct axial direction, avoiding poor thread engagement or jamming caused by axial misalignment.

[0134] Step S180-5-3: Control the robot end effector to drive the target fastener to feed and rotate along the tightening axis direction to perform the tightening action.

[0135] In this step, after the axis alignment is completed, the robot end effector carries the fastener and feeds it axially (i.e., pushes it downward) along the tightening axis direction. At the same time, it controls the fastener to rotate around the tightening axis direction, so that the thread of the fastener gradually engages and screws into the thread of the mating part.

[0136] In some embodiments, during the axial feed insertion process, a disturbance signal is actively applied in the radial direction of the target fastener, causing the target fastener to oscillate in the radial direction. This disturbance simulates the hand tremor or tentative movement during manual tightening, allowing the end of the bolt or screw to automatically find the starting engagement point of the thread during the oscillation, thereby eliminating rigid jamming caused by residual errors in visual calibration and mechanical repetitive positioning errors.

[0137] In some embodiments, during the tightening action, the torque feedback of the tightening motor can be monitored in real time to determine the thread engagement state and the degree of tightening completion. Specifically, the torque feedback value of the tightening drive component and / or the axial displacement of the target fastener are acquired in real time. When the torque feedback value exceeds a preset torque threshold or the axial displacement does not reach a preset displacement target, it indicates that thread misalignment, stripping, foreign object jamming, or other abnormal conditions may occur during the tightening process. At this time, the robot end effector is controlled to exit the current insertion position and re-execute the alignment step of step S180-5-2 and the insertion and tightening step of this step. If the retry is successful, the tightening continues; if multiple retry attempts fail, an abnormal alarm is triggered, prompting manual intervention. Through the above real-time monitoring and abnormal recovery mechanism, this application can effectively cope with various sudden abnormalities during fully automated operation, reduce production line downtime caused by a single failure, and further improve the robustness and unmanned operation capability of the system.

[0138] The technical solution of this embodiment generates multiple candidate grasping poses and performs collision detection, enabling the system to autonomously select a safe grasping path in complex work scenarios. This avoids work failures or equipment collisions caused by interference between the grasping pose and environmental objects such as fixtures. When all preset candidate poses are infeasible, the grasping direction is resampled around the axis, allowing for the rapid generation of a new obstacle avoidance scheme without re-identifying the fastener pose. This effectively improves the system's adaptability in narrow or cluttered work spaces. After successful grasping, the fastener is moved to the target tightening position and aligned with the axis, providing accurate positioning for subsequent threaded assembly. The tightening action, which involves feeding and rotating along the axis, achieves the threaded connection between the fastener and the mating part, completing the entire autonomous assembly process.

[0139] This application also provides a robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making, referring to... Figure 2 As shown, Figure 2 This is a schematic diagram of a robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making, provided in an embodiment of this application. The system includes: Point cloud acquisition module 210 is used to acquire three-dimensional point cloud data of the work scene; Point cloud segmentation module 220 is used to segment the target point cloud data of the target fastener from the three-dimensional point cloud data; The pose estimation module 230 is used to input the target point cloud data into a trained point cloud feature extraction network to regress and predict the spatial position of a predetermined key point on the target fastener; and to determine the six-degree-of-freedom pose of the target fastener based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener. The attitude determination module 240 is used to determine the attitude category of the target fastener based on the six-degree-of-freedom pose, wherein the attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping. The robot control module 250 is used to determine the current task type to be performed based on the posture category, wherein the task type includes grasping task and posture adjustment task; when it is determined that the posture adjustment task should be performed, the robot end effector is controlled to perform the posture adjustment task on the target fastener, and after the posture adjustment task, the three-dimensional point cloud data of the target fastener is reacquired to update the six degrees of freedom pose and the posture category; when it is determined that the grasping task should be performed, the robot end effector is controlled to grasp the target fastener based on the six degrees of freedom pose and perform a tightening operation.

[0140] In an optional embodiment, the point cloud segmentation module 220 is further configured to: The three-dimensional point cloud data is spatially filtered according to preset spatial range parameters to filter out background point clouds outside the area where the target fastener is located. Outlier removal is performed on the filtered point cloud data; Clustering and segmentation of the point cloud data after filtering out outliers is performed based on the point cloud density distribution to obtain the target point cloud of the target fastener.

[0141] In an optional embodiment, the point cloud segmentation module 220 is further configured to: Simulated light rays are projected from a preset viewpoint onto each point in the filtered point cloud data; If there are other point cloud occlusions on the path connecting the current point and the preset viewpoint position, then the target point is retained; If there are no other point cloud occlusions on the path connecting the current point and the preset viewpoint, the target point is identified as an outlier and removed.

[0142] In an optional embodiment, the system further includes a training module for training the point cloud feature extraction network according to the following steps: Construct a training dataset, wherein each training sample in the training dataset includes three-dimensional point cloud data of the sample fastener and corresponding annotation information, wherein the annotation information includes the true coordinates of multiple predetermined key points on the sample fastener and the true height of the sample fastener; The three-dimensional point cloud data of the sample fastener is input into the point cloud feature extraction network to be trained to obtain the predicted coordinates of the multiple predetermined key points and the predicted height of the sample fastener. Calculate the key point loss based on the difference between the predicted coordinates and the actual coordinates of the multiple predetermined key points; Based on the spatial collinearity relationship between the predicted coordinates of the multiple predetermined key points, the direction loss is calculated. Calculate the size loss based on the difference between the predicted height and the actual height; The network parameters of the point cloud feature extraction network to be trained are updated based on the key point loss, the orientation loss, and the size loss.

[0143] In an optional embodiment, the predetermined key points include a first key point, a second key point, and a third key point, wherein the first key point, the second key point, and the third key point correspond to the top feature position, the middle feature position, and the bottom feature position of the target fastener, respectively; the pose estimation module 230 is further configured to: The feature point type corresponding to each of the predetermined key points is determined according to the standard model, and the feature point type includes top feature point, middle feature point and bottom feature point; Based on the feature point type, determine the positions of the first key point, the second key point, and the third key point from the spatial positions of the predetermined key points; A first direction vector is determined based on the positions of the first key point and the second key point, and a second direction vector is determined based on the positions of the second key point and the third key point. Based on the average direction of the first direction vector and the second direction vector, the axial direction of the target fastener is determined, and the spatial position of the key point corresponding to the top feature point of the standard model among the predetermined key points is determined as the origin of the local coordinate system. A local coordinate system for the target fastener is constructed based on the axial direction and the origin, and the six-degree-of-freedom pose is determined based on the rotation and displacement relationships of the local coordinate system relative to the reference coordinate system; wherein, the reference coordinate system is the coordinate system in which the three-dimensional point cloud data is located.

[0144] In an optional embodiment, the attitude determination module 240 is further configured to: The central axis direction vector of the target fastener is extracted from the six-degree-of-freedom pose, and the central axis direction vector points from the bottom end to the top end of the target fastener; Obtain the gravity direction vector and calculate the angle between the central axis direction vector and the gravity direction vector; When the included angle is less than or equal to the first preset angle threshold, the posture category is determined to be the first posture category, which indicates that the target fastener is in a posture with the top facing down. When the included angle is greater than the first preset angle threshold and less than the second preset angle threshold, the posture category is determined to be the second posture category; the second posture category indicates that the target fastener is in a side-lying posture, and the first preset angle threshold is less than the second preset angle threshold; When the included angle is greater than or equal to the second preset angle threshold, the posture category is determined to be the third posture category, which indicates that the target fastener is in a posture with its top facing upwards.

[0145] In an optional embodiment, the robot control module 250 is further configured to: When the posture category is the first posture category or the third posture category, it is determined that the grasping operation should be performed at this time; When the attitude category is the second attitude category, it is determined that the attitude adjustment operation should be performed.

[0146] In an optional embodiment, the robot control module 250 is further configured to: One or more candidate gripping poses for the target fastener are generated based on the six-degree-of-freedom pose. Collision detection is performed between each of the candidate grasping poses and the 3D point cloud data of the current working scene; When a collision-free candidate grasping pose exists, the robot end effector is controlled to grasp the target fastener according to the collision-free candidate grasping pose. When all candidate grasping poses have a collision risk, the grasping direction is resampled around the axis direction of the target fastener as a constraint to generate a new candidate grasping pose. The collision detection is then re-executed until a collision-free grasping pose is obtained, and then the robot end effector is controlled to grasp the target fastener. Move the target fastener to the target tightening position and perform the tightening operation.

[0147] In an optional embodiment, the robot control module 250 is further configured to: Based on the six-degree-of-freedom pose of the target fastener, determine the target tightening position and the direction of the tightening axis; The robot's end effector is controlled to move the grasped target fastener to the target tightening position and align the axis of the target fastener with the tightening axis. The robot's end effector drives the target fastener to feed and rotate along the tightening axis, performing a tightening action.

[0148] It is understood that the robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making in the embodiments of this application can realize the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making in the above embodiments. The robot autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making has the same advantages as the above-mentioned robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making compared with the prior art, and will not be repeated here.

[0149] This application also provides an electronic device, which is described in reference to... Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 300 includes a memory 310 and a processor 320. The memory 310 and the processor 320 are connected via a bus for communication. The memory 310 stores a computer program that can run on the processor 320 to implement the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making described in the embodiments of this application.

[0150] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making described in this application.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making described in this application.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0153] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0156] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0157] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0158] The above provides a detailed description of a robot autonomous tightening method and system based on 3D point cloud key point detection and adaptive decision-making. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A robot autonomous tightening method based on 3D point cloud key point detection and adaptive decision-making, characterized in that, include: Acquire 3D point cloud data of the work scene; The target point cloud data of the target fastener is segmented from the three-dimensional point cloud data; The target point cloud data is input into a trained point cloud feature extraction network to regress and predict the spatial location of predetermined key points on the target fastener. Based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener, the six-degree-of-freedom pose of the target fastener is determined; The attitude category of the target fastener is determined based on the six-degree-of-freedom pose, and the attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping; Based on the posture category, determine the type of task to be performed, which includes grasping tasks and posture adjustment tasks; When it is determined that the attitude adjustment operation should be performed, the robot end effector is controlled to perform the attitude adjustment operation on the target fastener, and the three-dimensional point cloud data of the target fastener is reacquired after the attitude adjustment operation to update the six-degree-of-freedom pose and the attitude category; When it is determined that the gripping operation should be performed, the robot end effector grips the target fastener according to the six-degree-of-freedom pose control and performs the tightening operation.

2. The method according to claim 1, characterized in that, Segmenting the target point cloud data of the target fastener from the three-dimensional point cloud data includes: The three-dimensional point cloud data is spatially filtered according to preset spatial range parameters to filter out background point clouds outside the area where the target fastener is located. Outlier removal is performed on the filtered point cloud data; Clustering and segmentation of the point cloud data after filtering out outliers is performed based on the point cloud density distribution to obtain the target point cloud of the target fastener.

3. The method according to claim 2, characterized in that, Outlier removal is performed on the filtered point cloud data, including: Simulated light rays are projected from a preset viewpoint onto each point in the filtered point cloud data; If there are other point cloud occlusions on the path connecting the current point and the preset viewpoint position, then the target point is retained; If there are no other point cloud obstructions on the path connecting the current point and the preset viewpoint, the target point is identified as an outlier and removed.

4. The method according to claim 1, characterized in that, The point cloud feature extraction network is trained according to the following steps: Construct a training dataset, wherein each training sample in the training dataset includes three-dimensional point cloud data of the sample fastener and corresponding annotation information, wherein the annotation information includes the true coordinates of multiple predetermined key points on the sample fastener and the true height of the sample fastener; The three-dimensional point cloud data of the sample fastener is input into the point cloud feature extraction network to be trained to obtain the predicted coordinates of the multiple predetermined key points and the predicted height of the sample fastener. Calculate the key point loss based on the difference between the predicted coordinates and the actual coordinates of the multiple predetermined key points; Based on the spatial collinearity relationship between the predicted coordinates of the multiple predetermined key points, the direction loss is calculated. Calculate the size loss based on the difference between the predicted height and the actual height; The network parameters of the point cloud feature extraction network to be trained are updated based on the key point loss, the orientation loss, and the size loss.

5. The method according to claim 1, characterized in that, The predetermined key points include a first key point, a second key point, and a third key point, wherein the first key point, the second key point, and the third key point correspond to the top feature position, the middle feature position, and the bottom feature position of the target fastener, respectively. Based on the transformation relationship between the spatial positions of the predetermined key points and the positions of the corresponding key points on the standard model of the target fastener, the six-degree-of-freedom pose of the target fastener is determined, including: The feature point type corresponding to each of the predetermined key points is determined according to the standard model, and the feature point type includes top feature point, middle feature point and bottom feature point; Based on the feature point type, determine the positions of the first key point, the second key point, and the third key point from the spatial positions of the predetermined key points; A first direction vector is determined based on the position of the first key point and the position of the second key point, and a second direction vector is determined based on the position of the second key point and the position of the third key point. Based on the average direction of the first direction vector and the second direction vector, the axial direction of the target fastener is determined, and the spatial position of the key point corresponding to the top feature point of the standard model among the predetermined key points is determined as the origin of the local coordinate system. A local coordinate system for the target fastener is constructed based on the axial direction and the origin, and the six-degree-of-freedom pose is determined based on the rotation and displacement relationships of the local coordinate system relative to the reference coordinate system; wherein, the reference coordinate system is the coordinate system in which the three-dimensional point cloud data is located.

6. The method according to claim 1, characterized in that, Determining the attitude category of the target fastener based on the six-degree-of-freedom pose includes: The central axis direction vector of the target fastener is extracted from the six-degree-of-freedom pose, and the central axis direction vector points from the bottom end to the top end of the target fastener; Obtain the gravity direction vector and calculate the angle between the central axis direction vector and the gravity direction vector; When the included angle is less than or equal to the first preset angle threshold, the posture category is determined to be the first posture category, which indicates that the target fastener is in a posture with the top facing down. When the included angle is greater than the first preset angle threshold and less than the second preset angle threshold, the posture category is determined to be the second posture category; the second posture category indicates that the target fastener is in a side-lying posture, and the first preset angle threshold is less than the second preset angle threshold; When the included angle is greater than or equal to the second preset angle threshold, the posture category is determined to be the third posture category, which indicates that the target fastener is in a posture with its top facing upwards.

7. The method according to claim 6, characterized in that, Based on the attitude category, determine the type of task to be performed, including: When the posture category is the first posture category or the third posture category, it is determined that the grasping operation should be performed at this time; When the attitude category is the second attitude category, it is determined that the attitude adjustment operation should be performed.

8. The method according to claim 1, characterized in that, The six-degree-of-freedom pose-controlled robot end effector grasps the target fastener and performs a tightening operation, including: One or more candidate gripping poses for the target fastener are generated based on the six-degree-of-freedom pose. Collision detection is performed between each of the candidate grasping poses and the 3D point cloud data of the current working scene; When a collision-free candidate grasping pose exists, the robot end effector is controlled to grasp the target fastener according to the collision-free candidate grasping pose. When all candidate grasping poses have a collision risk, the grasping direction is resampled around the axis direction of the target fastener as a constraint to generate a new candidate grasping pose. The collision detection is then re-executed until a collision-free grasping pose is obtained, and then the robot end effector is controlled to grasp the target fastener. Move the target fastener to the target tightening position and perform the tightening operation.

9. The method according to claim 8, characterized in that, Moving the target fastener to the target tightening position and performing the tightening operation includes: Based on the six-degree-of-freedom pose of the target fastener, determine the target tightening position and the direction of the tightening axis; The robot's end effector is controlled to move the grasped target fastener to the target tightening position and align the axis of the target fastener with the tightening axis. The robot's end effector drives the target fastener to feed and rotate along the tightening axis, performing a tightening action.

10. A robotic autonomous tightening system based on 3D point cloud key point detection and adaptive decision-making, characterized in that, include: The point cloud acquisition module is used to acquire 3D point cloud data of the work scene; The point cloud segmentation module is used to segment the target point cloud data of the target fastener from the three-dimensional point cloud data; The pose estimation module is used to input the target point cloud data into a trained point cloud feature extraction network to regress and predict the spatial position of a predetermined key point on the target fastener; and to determine the six-degree-of-freedom pose of the target fastener based on the transformation relationship between the spatial position of the predetermined key point and the position of the corresponding key point on the standard model of the target fastener. An attitude determination module is used to determine the attitude category of the target fastener based on the six-degree-of-freedom pose. The attitude category is used to characterize whether the current spatial orientation of the target fastener is suitable for direct grasping. The robot control module is used to determine the type of task to be performed based on the posture category, wherein the task type includes grasping tasks and posture adjustment tasks; When it is determined that the attitude adjustment operation should be performed, the robot end effector is controlled to perform the attitude adjustment operation on the target fastener, and the three-dimensional point cloud data of the target fastener is reacquired after the attitude adjustment operation to update the six-degree-of-freedom pose and the attitude category; When it is determined that the gripping operation should be performed, the robot end effector grips the target fastener according to the six-degree-of-freedom pose control and performs the tightening operation.

Images