Industrial bolt intelligent identification and precise grabbing control method and system based on depth vision

Abstract

The invention discloses an industrial bolt intelligent identification and accurate grabbing control method and system based on depth vision, and the method comprises the steps: obtaining multispectral image data, carrying out the fusion detection of the nut edge, thread texture and other features of a bolt through an enhanced YOLOv8 network, screening an effective bolt, and eliminating the interference of a gasket, chippings and the like; extracting feature points to solve a 6DoF attitude, fusing vibration data of the mechanical arm to realize motion compensation, and outputting a three-dimensional coordinate and an attitude angle under a coordinate system of the mechanical arm; the three-dimensional coordinates are converted into motion coordinates, and the grabbing position is corrected in combination with the normal vector of the assembly surface; planning a path and generating grabbing strategies such as clamping jaw opening amplitude and clamping force matched with the bolt specification; the clamping force is dynamically adjusted based on feedback of the six-dimensional force sensor, and accurate grabbing is achieved. The system comprises a visual perception subsystem, an intelligent decision-making subsystem and a precise execution subsystem. The bolt recognition precision and grabbing stability in the complex environment are improved, and the assembly requirements of bolts of multiple specifications can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of program-controlled manipulators, and in particular to an industrial bolt intelligent recognition and precise grasping control method and system based on depth vision. BACKGROUND

[0002] Currently, industrial bolt assembly is a key link in the field of industrial automation, and its precision and efficiency directly affect the structural stability and production quality of products. With the upgrading of manufacturing industry to intelligence, higher requirements are put forward for the automation degree, adaptability and precision of bolt assembly. As a frontier technology integrating computer vision and deep learning, depth vision technology can realize precise recognition, positioning and pose estimation of targets through depth analysis and feature extraction of images, providing core perception ability for automatic operation in complex industrial scenes.

[0003] Industrial bolt assembly technology has undergone three generations of development: the first generation (2022-2023) is pure mechanical positioning, relying on precision clamps and positioning pins, and is realized by special assembly machines, but the system has poor flexibility and needs to be re-adjusted when changing bolt specifications, which cannot adapt to diversified production; the second generation (2023-2024) introduces basic vision assistance, using traditional image processing algorithms such as OpenCV for feature matching, but its performance depends on manually designed features, is sensitive to changes in lighting, and the recognition rate decreases significantly when the bolt is partially obscured, and the positioning accuracy is limited by image resolution; the third generation (2024-today) starts to apply deep learning technology, such as CNN target detection combined with servo control, but the existing control system still has obvious defects, mostly using pulse control, which has weak anti-interference ability, poor synchronization when multiple axes are coordinated, and lacks real-time torque feedback, making it difficult to ensure the consistency of assembly quality.

[0004] Overall, existing technologies have limitations in environmental adaptability, recognition accuracy, and control stability, and cannot meet the needs of high-precision and high-flexibility industrial bolt assembly.

[0005] Therefore, it is necessary to improve the existing technology of industrial bolt intelligent recognition and precise grasping control method and system based on depth vision to solve the above problems. SUMMARY

[0006] The present application overcomes the shortcomings of the prior art and provides an industrial bolt intelligent recognition and precise grasping control method and system based on depth vision, aiming to solve the problems of poor adaptability to complex environments, insufficient positioning accuracy, and insufficient adaptability to multi-specification bolts in the prior art.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows: an industrial bolt intelligent recognition and precise grasping control method based on depth vision, comprising:

[0008] S1. Acquire multispectral image data of industrial bolts, and use an enhanced YOLOv8 network to fuse and detect the feature data of industrial bolts in the multispectral images, filter out valid bolts, and output target data; among which, the feature data of industrial bolts include nut edge data and thread texture data;

[0009] S2. Extract image feature point data from the target data of the effective bolts, calculate the 6DoF attitude of the bolts in the camera coordinate system, perform motion compensation, and convert the attitude data to the robot arm coordinate system through the hand-eye calibration matrix. Output the three-dimensional coordinates and attitude angle data of the industrial bolts in the robot arm coordinate system.

[0010] S3. Convert the three-dimensional coordinate data in the robotic arm coordinate system into motion coordinates that the robotic arm can recognize, and combine them with the normal vector data of the assembly surface to correct the target position for gripping industrial bolts.

[0011] S4. Plan the motion path of the robotic arm from its current position to the industrial bolt gripping point, and generate gripping strategy data based on the bolt specifications; the gripping strategy data includes gripper opening amplitude data and gripping force parameter data.

[0012] S5. The robotic arm is controlled by path data and grasping strategy data. Based on the contact force data fed back by the six-dimensional force sensor, the clamping force and torque are dynamically adjusted to accurately grasp industrial bolts.

[0013] In a preferred embodiment of the present invention, in step S1, the enhanced YOLOv8 network is improved based on the YOLOv8 network, including: embedding a CBAM convolutional attention module in the neck network to enhance the edge and thread texture features of the nut from the channel and spatial dimensions; using a BiFPN feature fusion structure to construct a bidirectional transmission path, fusing small-scale detail features and large-scale global features; and setting the anchor parameters for small target detection to 5×6, 8×10, and 12×16, respectively adapted to M6-M7, M8-M9, and M10-M11 bolts.

[0014] In a preferred embodiment of the present invention, in step S1, the multispectral image data includes visible light and near-infrared dual-channel images; the visible light channel has a wavelength of 400-700nm, and the near-infrared channel has a wavelength of 850nm; the illumination angle is dynamically adjusted from 0° to 180° by the polarization illumination direction, and the polarization direction is automatically rotated to be perpendicular to the reflection direction when specular reflection on the bolt surface is detected; wherein, the visible light channel is used to capture the bolt color and surface texture features, and the near-infrared channel is used to penetrate oil stains and rust interference to collect the underlying structural features of the bolt.

[0015] In a preferred embodiment of the present invention, step S2, the 6DoF attitude calculation includes: extracting the three-dimensional coordinates of the nut feature points and the three-dimensional coordinates of the thread feature points; matching the feature point set with a preset feature library of three-dimensional bolt models of the same specification; calculating the initial attitude using the PnP algorithm; and iteratively optimizing the position accuracy and attitude angle accuracy using the LM algorithm.

[0016] In a preferred embodiment of the present invention, step S2, which integrates the vibration data of the robotic arm to achieve motion compensation, includes: using an inertial measurement unit to collect the vibration acceleration and angular velocity of the robotic arm, calculating the position offset and angular offset through Kalman filtering, and performing real-time correction on the six-degree-of-freedom attitude.

[0017] In a preferred embodiment of the present invention, correcting the target position by combining normal vector data with a compensation algorithm includes:

[0018] The point cloud data of the assembly surface is collected, and the normal vector of the assembly surface in the robot arm coordinate system is calculated by the plane fitting algorithm.

[0019] The compensation coefficient is dynamically determined based on the specifications of the industrial bolts. The compensation coefficient is used to control the reserved safety clearance between the clamp and the assembly surface.

[0020] The initial gripping target position is spatially offset and corrected along the normal vector direction so that the corrected gripping target position satisfies that the contact direction of the robotic arm end effector is perpendicular to the assembly surface.

[0021] In a preferred embodiment of the present invention, step S4, path planning includes:

[0022] A fifth-order polynomial interpolation algorithm is used to generate collision-free trajectories, and a maximum acceleration and deceleration mechanism when approaching the target are set. The spacing between path nodes is also dynamically adjusted based on the bolt specifications.

[0023] A fifth-order polynomial function is constructed to describe the positional change of the robotic arm's end effector in the time dimension. The position coordinates, velocity, and acceleration of the starting and ending points are input as boundary conditions to determine the polynomial function. The position and attitude of the robotic arm's end effector are calculated at fixed time intervals to generate continuous intermediate nodes. When the trajectory enters a prohibited area, the polynomial coefficients are fine-tuned to correct the path.

[0024] Set the maximum acceleration to 2 m / s². 2 When the end of the robotic arm is 50mm away from the target position, a deceleration mechanism is triggered; the path node spacing is dynamically adjusted based on the bolt specifications, with the node spacing of small bolts (M6-M11) ≤ 5mm and the node spacing of large bolts (M12-M20) ≤ 10mm.

[0025] In a preferred embodiment of the present invention, step S4, generating grabbing strategy data adapted to industrial bolt specifications, includes:

[0026] The initial opening width of the jaws is determined based on the standard diagonal dimensions of industrial bolts and nuts. The initial opening width is the sum of the standard diagonal dimensions of the nut and the safety margin.

[0027] The clamping force parameters are set according to the specifications of industrial bolts. The clamping force for small bolts (M6-M11) is 0.5-1.5N, of which 1.0-1.5N is used for carbon steel bolts and 0.5-1.0N is used for stainless steel bolts. The clamping force for large bolts (M12-M20) is 2.0-4.0N, and increases with the bolt size.

[0028] By combining the industrial bolt attitude angle to correct the clamping force parameters, the clamping jaw closing process is simulated in a virtual environment to check the distribution of contact points between the clamping jaw and the nut. If there is contact offset, the clamping jaw attitude angle is finely adjusted within ±1° and the opening amplitude and clamping force are recalculated.

[0029] In a preferred embodiment of the present invention, step S5, in which the robotic arm dynamically adjusts the clamping force and torque based on the contact force data fed back by the six-dimensional force sensor, includes:

[0030] The path data and grasping strategy data are transmitted to the robotic arm controller through the CANopen communication module. The CANopen communication module adopts a dual-channel redundant design with a baud rate of 1Mbps and a synchronization period of 1ms.

[0031] After parsing the data, the robotic arm controller generates motion commands for each joint, which are then executed by the servo drive array. The servo drive array uses three levels of adjustment—position loop, speed loop, and current loop—to make the end effector of the robotic arm move along the planned trajectory.

[0032] The six-dimensional force sensor and strain gauge of the intelligent gripper at the end of the robotic arm synchronously collect contact force and rotational torque signals. Based on these signals, the robotic arm controller dynamically adjusts the gripping force and rotational torque through an adaptive gripping force control algorithm and a real-time torque control algorithm.

[0033] This invention provides an intelligent recognition and precise gripping control system for industrial bolts based on depth vision, comprising a visual perception subsystem, an intelligent decision-making subsystem, and a precise execution subsystem:

[0034] The visual perception subsystem includes: a multispectral imaging module, an enhanced YOLOv8 detection network module, and a 3D pose calculation module. The multispectral imaging module is used to acquire multispectral image data of the environment where the industrial bolts are located. The enhanced YOLOv8 detection network module is used to fuse and detect the feature data of the industrial bolts in the multispectral images and select valid bolts. The 3D pose calculation module is used to extract image feature point data, solve 6DoF pose, and fuse robotic arm vibration data to achieve motion compensation.

[0035] The intelligent decision-making subsystem includes: a spatial coordinate mapping module, a motion trajectory planning module, and a grasping strategy optimization module. The spatial coordinate mapping module is used to convert three-dimensional coordinate data into motion coordinates that the robotic arm can recognize and to correct the position of the grasping target by combining normal vector data. The motion trajectory planning module is used to plan the path data of the robotic arm from the current position to the grasping point. The grasping strategy optimization module is used to generate grasping strategy data that is adapted to the specifications of industrial bolts.

[0036] The precision execution subsystem includes: a CANopen communication module, a servo drive array, and an intelligent gripper module; the CANopen communication module is used to transmit path data and gripping strategy data, the servo drive array is used to execute joint motion commands to make the robotic arm move along the planned trajectory, and the intelligent gripper module is used to dynamically adjust the gripping force and torque based on the contact force data fed back by the six-dimensional force sensor to achieve precise gripping.

[0037] This invention addresses the shortcomings of the prior art and has the following beneficial effects:

[0038] (1) This invention uses multispectral image data combined with an enhanced YOLOv8 network for bolt recognition, embeds a CBAM convolutional attention module to enhance the edge and thread texture features of the nut, and also adopts a BiFPN feature fusion structure and sets dedicated anchor parameters, so that the system can capture the surface texture and underlying structural features of the bolt, accurately distinguish the bolt from interference objects such as gasket debris, and adapt to the detection requirements of bolts of different specifications. It achieves stable bolt recognition in complex environments. Compared with the existing technology, which is sensitive to changes in light and has a low recognition rate, especially for small bolts, it can stably recognize bolts in scenarios with sudden changes in light and surface contamination.

[0039] (2) When calculating the 6DoF attitude of the bolt, this invention extracts the three-dimensional feature points of the nut and thread and matches them with a preset model library, and integrates robotic arm vibration data compensation; it comprehensively acquires the spatial position and attitude information of the bolt, while eliminating the error caused by robotic arm vibration, thus achieving high-precision bolt positioning. Existing technologies only output two-dimensional positions and do not consider the influence of vibration, resulting in grasping offset. This system further solves the problem of visual positioning and actual grasping deviation through three-dimensional attitude calculation and vibration compensation, providing a reliable position benchmark for accurate grasping.

[0040] (3) When determining the target position of the robotic arm, the present invention combines the normal vector data of the assembly surface where the industrial bolt is located and corrects the position through a compensation algorithm so that the contact direction of the end effector of the robotic arm can adapt to the tilt state of the assembly surface. This avoids the problem that the existing technology does not consider the influence of the normal vector of the assembly surface, which often results in accurate visual positioning but actual grasping deviation.

[0041] (4) The present invention adopts a fifth-order polynomial interpolation algorithm, sets a deceleration mechanism and dynamically adjusts the node spacing according to the bolt specifications, and generates a gripping strategy that adapts to the specifications, including the gripper opening amplitude and gripping force, so that the robotic arm moves smoothly and without collision, and the gripping parameters are accurately matched with the bolt specifications, thereby improving the safety and reliability of the gripping process.

[0042] (5) When controlling the robotic arm, the present invention dynamically adjusts the clamping force and torque based on the contact force data fed back by the six-dimensional force sensor, forming a closed-loop regulation. It responds in real time to force changes during the grasping process, avoiding the problem of bolt slippage due to excessively loose clamping or bolt damage due to excessively tight clamping. Existing technologies lack real-time torque feedback, making it difficult to ensure the consistency of assembly quality. Through force control adjustment, the stability of grasping quality is further ensured. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a preferred embodiment of the present invention;

[0045] Figure 2 This is a preferred embodiment of the multispectral image acquisition execution diagram of the present invention;

[0046] Figure 3 This is a diagram of an industrial bolt identification machine according to a preferred embodiment of the present invention;

[0047] Figure 4 This is a performance monitoring curve during the training process of the YOLOv8 model, which is a preferred embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein. Therefore, the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0050] Application Overview:

[0051] This application targets bolts of various specifications for harsh working conditions, including: outdoor oily environments of power towers, metallic reflective scenarios of heavy machinery, and automated production lines and intelligent equipment for densely obscured assembly surfaces of automobile chassis.

[0052] The core contradiction in these scenarios lies in the fact that bolt assembly needs to remain stable under dynamic conditions such as sudden changes in lighting, surface contamination, and random workpiece posture. Bolts are often on inclined planes or slightly offset due to vibration. Traditional solutions only output the position of two-dimensional images and ignore the influence of the normal vector of the assembly surface. When the bolt is on an inclined plane or slightly offset due to vibration, it leads to the contradiction that the visual positioning is accurate but the actual grasping is off.

[0053] Existing technologies treat interferences such as light, oil stains, and occlusion as problems that can be solved by increasing image resolution or algorithm parameters, but fail to realize that the essence of these interferences is the dynamic change of the optical properties of the bolt surface.

[0054] Therefore, traditional solutions, which rely on single visible light imaging or passive illumination, cannot adapt to the dynamic fluctuations in optical properties. The design logic of existing control systems is to execute according to a preset trajectory, but when coordinating multiple axes, traditional technologies lack real-time torque feedback and cannot adjust the force according to the changes in force during the bolt tightening process.

[0055] This application utilizes multispectral imaging to fuse visible light and near-infrared features, adapting to the dynamic changes in the optical properties of bolt surfaces; it introduces assembly surface normal vector calculation and vibration compensation to solve the deviation problem between two-dimensional positioning and three-dimensional grasping; and it combines dual-channel redundant communication and force-torque dual closed-loop adjustment to achieve high precision and real-time response of multi-axis collaboration.

[0056] Exemplary method:

[0057] like Figure 1 As shown, the method for intelligent identification and precise grasping control of industrial bolts based on depth vision includes:

[0058] S1. Acquire multispectral image data of industrial bolts, and use an enhanced YOLOv8 network to fuse and detect the feature data of industrial bolts in the multispectral images, filter out valid bolts, and output target data; among which, the feature data of industrial bolts include nut edge data and thread texture data;

[0059] S2. Extract image feature point data from the target data of the effective bolts, calculate the 6DoF attitude of the bolts in the camera coordinate system, perform motion compensation, and convert the attitude data to the robot arm coordinate system through the hand-eye calibration matrix. Output the three-dimensional coordinates and attitude angle data of the industrial bolts in the robot arm coordinate system.

[0060] S3. Convert the three-dimensional coordinate data in the robotic arm coordinate system into motion coordinates that the robotic arm can recognize, and combine them with the normal vector data of the assembly surface to correct the target position for gripping industrial bolts.

[0061] S4. Plan the motion path of the robotic arm from its current position to the industrial bolt gripping point, and generate gripping strategy data based on the bolt specifications; the gripping strategy data includes gripper opening amplitude data and gripping force parameter data.

[0062] S5. The robotic arm is controlled by path data and grasping strategy data. Based on the contact force data fed back by the six-dimensional force sensor, the clamping force and torque are dynamically adjusted to accurately grasp industrial bolts.

[0063] In the process of intelligent identification and precise grasping of industrial bolts, it is first necessary to obtain reliable bolt target information through a vision system to provide a data foundation for subsequent detection and grasping.

[0064] In step S1, the multispectral imaging module and the enhanced YOLOv8 detection network are started. For the industrial bolt scenario, the visible light and near-infrared dual-channel image acquisition parameters, polarization illumination direction, and special anchor parameters adapted to bolts of various specifications are configured to complete the system initialization.

[0065] Industrial bolts refer to metal fasteners used to connect and fix workpieces in industrial assembly. Their forms include nuts, bolts, and threaded structures, and their specifications cover common models of M6 and above. In scenarios such as outdoor oily environments of power towers, metal reflection scenes of heavy machinery, and densely obscured assembly surfaces of automobile chassis, bolt connection areas are subject to complex environmental factors such as changes in lighting, surface oil, and workpiece obstruction.

[0066] like Figure 2 , Figure 3 As shown, multispectral image data refers to image data that simultaneously contains information from the visible and near-infrared bands, reflecting the optical characteristics of industrial bolts under different spectra; visible and near-infrared dual-channel image acquisition refers to the settings for acquiring images using two bands. The visible light channel is used to capture the conventional features of the bolt's color and surface texture, with a wavelength of 400-700nm; the near-infrared channel is used to penetrate the interference features of oil stains and slight rust on the bolt surface to acquire deeper structural features, with a wavelength of 800-950nm.

[0067] Preferably, the near-infrared wavelength is 850nm. This wavelength has strong penetrability and can penetrate the slight oil stains, dust, oxide layers and other interference on the surface of industrial bolts. At the same time, it has moderate reflectivity to metal materials and can clearly present the underlying structural features of the bolt, including: the spiral pattern of the thread and the edge contour of the nut. In S1, the image features of visible light and near-infrared light are combined to improve the detection accuracy of the industrial bolt surface through feature fusion.

[0068] Polarized illumination direction refers to the parameter used to adjust the polarization angle of the illumination light. By changing the polarization direction, the interference of specular reflection on the bolt surface, especially metal materials, can be reduced, adapting to imaging requirements under different lighting conditions. Dedicated anchor parameters adapted to multiple bolt specifications refer to industrial bolts of different sizes and head shapes, including industrial bolts of models M6-M20 with hexagonal or round heads. In the detection network, preset feature matching box parameters improve the detection targeting of industrial bolts of different specifications.

[0069] The system dynamically adjusts the angle of polarized illumination according to the lighting conditions of the industrial bolt scene, and the illumination angle is continuously adjustable from 0° to 180°. When specular reflection is detected on the bolt surface, the polarization direction is automatically rotated to be perpendicular to the reflection direction to reduce the strong reflection interference of the metal surface. In low light or shadow environments, the polarization angle is adjusted to maximize light utilization and enhance the contrast between bolt features and background.

[0070] Specifically, the lighting parameters are as follows: resolution of 3840×2700-4096×3000 to ensure that small bolts can still present sufficient pixel details in the image; frame rate of 20-25fps@full resolution to meet the real-time capture of bolt movement in dynamic scenes; dynamic range of 130-140dB to clearly present the highlight and shadow areas on the bolt surface at the same time, avoiding feature loss due to uneven lighting.

[0071] like Figure 4 As shown, the enhanced YOLOv8 detection network refers to a target detection model optimized for the characteristics of power grid bolts. It improves the bolt recognition capability in small-sized and high-interference scenarios by improving the network structure.

[0072] The original structure of YOLOv8 consists of three parts:

[0073] The backbone network is used to extract basic features from the two-channel image, including the edges and texture of the bolt;

[0074] Neck network is used to fuse features at different scales, including the overall outline of large targets and the details of small targets;

[0075] Head network is used to output the location and category of the target based on features.

[0076] The enhanced YOLOv8 incorporates improvements to the original structure and optimizes parameters to enhance the inspection accuracy of industrial bolts, especially small-diameter bolts.

[0077] Specifically, the improvements made to the enhanced YOLOv8 detection network based on the original YOLOv8 include:

[0078] The neck network is a key part of the network responsible for feature integration and transmission. Embedding the CBAM convolutional attention module allows the network to more accurately focus on bolt features.

[0079] The CBAM convolutional attention module is an algorithmic component that enables the enhanced YOLOv8 network to actively focus on the key features of bolts and ignore irrelevant interference. By simulating the human visual attention mechanism for key areas, it improves the extraction accuracy of bolt features in multispectral images.

[0080] To address the issue that bolts in industrial settings are often obscured by oil stains, washers, or are similar in color / material to the background, the CBAM module enhances bolt feature signals through a dual-channel attention mechanism:

[0081] First, determine the image channel or feature that best represents the bolt based on the channel dimension;

[0082] Then, locate the possible areas where the bolts may exist in the image from a spatial perspective;

[0083] Ultimately, by allocating weights, the network prioritizes the core features of the bolt, including the nut edge and thread texture, while minimizing background interference, such as oil stains and workpiece surface scratches.

[0084] Specifically, the operating logic of the CBAM convolutional attention module is as follows:

[0085] The input data consists of preliminary feature maps extracted from multispectral images using Backbone, including: color and texture features of the visible light channel, structural features of the near-infrared channel, and preliminary features of the nut edge and thread texture;

[0086] First, filter key feature channels by channel dimension. Input the preliminary feature map of the multispectral image, which contains multiple channels. Each channel corresponds to a feature. Perform global pooling on the feature map of each channel to represent the feature intensity of that channel.

[0087] The importance weights of different channels are learned through a fully connected layer; in one embodiment, the weights of the nut edge channel and the thread texture channel are increased because they are more critical for bolt identification, while the weights of channels affected by oil are decreased.

[0088] By multiplying the channel features by their own weights, the characteristic signals of key channels are amplified, while interfering channels are suppressed.

[0089] Then, the area where the bolt is located is located by spatial dimension, and the feature map after channel attention weighting is input. At this time, the key channels have been enhanced.

[0090] Pooling is performed on the feature map along the channel dimension, and the maximum and average feature values ​​at each pixel location are taken to obtain two spatial feature maps, which reflect the local maximum response and the global average response, respectively.

[0091] After concatenating the two spatial feature maps, spatial weights are learned through a convolutional layer. In this embodiment, because they conform to the geometric characteristics of a bolt, the weights of the hexagonal nut region and the spiral thread region are increased; the weights of the flat workpiece surface and the background region with irregular debris are decreased.

[0092] Multiplying the spatial weights by the spatial feature map enhances the features of the region where the bolt is located while weakening the background region.

[0093] The aforementioned CBAM convolutional attention module enables the focus on the unobstructed threaded area or nut edge through spatial weights, even if the bolt is partially obscured, ensuring that the target is not missed.

[0094] In the feature map processed by the CBAM module, the core features of the bolt, including the nut edge and thread texture, are enhanced in both the channel and spatial dimensions, making the feature signals more prominent; while the feature signals of interference such as oil stains and debris are suppressed.

[0095] Furthermore, the YOLOv8 detection network is improved using a BiFPN feature fusion structure. BiFPN feature fusion is an efficient cross-scale feature fusion structure specifically adapted to the diverse bolt specifications found in industrial scenarios. Located at the back end of the CBAM module in the Neck network, its core function is to integrate the features refined by CBAM.

[0096] BiFPN feature fusion establishes a bidirectional feature transfer path: it transfers small-scale detailed features extracted from the backbone to higher layers, and large-scale global features to lower layers, achieving deep fusion of bolt features of different specifications.

[0097] At the same time, BiFPN will prune redundant feature transfer links and retain only key fusion paths, which improves efficiency while ensuring that bolts as small as a few pixels as large bolts occupying a large area of ​​the image can be effectively identified.

[0098] The operational logic of BiFPN feature fusion:

[0099] The feature maps received after processing by the CBAM convolutional attention module have been enhanced with channel and spatial attention to enhance core features, including: nut edge and thread texture, and weaken the interference of oil stains and debris. These feature maps contain bolt features at different scales.

[0100] A bidirectional feature transfer path is constructed to transfer small-scale detail features enhanced by CBAM in the lower-level network to the higher-level network. Small-scale detail features include: the thread texture of small bolts and the subtle corners of the nut edges. Small-scale detail features supplement local details for large-scale features, ensuring that details such as the variation in thread depth of large bolts and wear marks on the nut surface are not ignored. Large-scale features include: the overall outline of large bolts.

[0101] Large-scale features enhanced by CBAM in the high-level network are passed to the low-level network. These features include the overall outline of the large bolt and its relative position with the surrounding workpieces. The large-scale features provide global context for the small-scale features, helping to distinguish the ownership of different small bolts in densely occluded scenes and avoiding misjudgment due to similar local features.

[0102] In industrial bolt inspection scenarios, the distinction between large-scale and small-scale features is primarily based on the spatial coverage of the features in the image and the granularity of the bolt information they describe. Specifically, this can be determined by combining bolt specifications and feature representation.

[0103] Small-scale features refer to features in an image that cover a small area and focus on the local details of the bolt. The corresponding bolt sizes are usually small, such as M6-M11 bolts, occupying only a few pixels in the image, ranging from 10 to 50 pixels. The core focus is on describing the local structural details of the bolt, including:

[0104] Thread texture, including the spacing and depth variation of the spiral patterns on the shank of the small bolt, is represented in the image as dense local grayscale fluctuations;

[0105] The subtle bends on the edge of the nut, including the angles of the hexagonal head of the small bolt and details of edge wear, are represented in the image as local line transitions.

[0106] These features can only reflect the local shape of the bolt and cannot fully describe the overall position of the bolt or its relationship with the surrounding environment.

[0107] Large-scale features refer to features in an image that cover a large area and contain global information about the bolt. Large-scale features typically correspond to large bolt sizes, such as M12-M20 bolts, occupying a significant number of pixels in the image (over 50 pixels). The core function is to describe the overall shape and spatial relationships of the bolt, including:

[0108] The overall outline of the large bolt includes: the overall geometry of the nut and the overall direction of the bolt, which is represented as a continuous large area in the image;

[0109] The relative position with surrounding workpieces includes the spatial arrangement of the large bolt with adjacent gaskets, flanges, and other workpieces, which is represented in the image as the positional relationship between the bolt and the surrounding area.

[0110] These features can reflect the overall shape and environmental context of the bolt, but lack local details.

[0111] After constructing the bidirectional feature transfer path, the feature links in the bidirectional transfer path are screened, and redundant links that contribute very little to bolt detection are pruned, retaining only the key fusion links.

[0112] Redundant links refer to feature transmission paths that contribute very little to bolt detection or may even introduce interference, including: pure background feature links, strong interference feature links, and low-confidence bolt feature links;

[0113] The pure background feature link is a feature that is transmitted entirely from the non-bolt area. Its features are characterized by no nut edge, no thread texture, uniform grayscale and irregular geometric shape.

[0114] Strong interference characteristic links transmit characteristics from interference targets, including gaskets, debris, or heavily contaminated bolt areas. Their characteristics are manifested as regular shapes without thread texture, irregular shapes without complete nut structures, and nut edges and thread textures completely covered by noise.

[0115] The low-confidence bolt feature link is where the transmitted bolt features are blurry, and information such as broken nut edges and lost thread texture due to insufficient lighting is transmitted. Even after processing by the CBAM module, the weight is still below the set threshold. Such features have extremely low support for bolt recognition, and continuing to transmit them will reduce fusion efficiency, so they need to be pruned.

[0116] Key fusion links refer to the transmission paths that directly contribute to bolt detection and enhance the complementarity of bolt features at different scales. These include: supplementary links from small-scale detail features to large-scale features, guiding links from large-scale global features to small-scale features, and cross-spectral feature synergy links.

[0117] The supplementary link from small-scale detailed features to large-scale features is that the small-scale features and large-scale features belong to the same assembly region, and the small-scale features can fill the detail gaps of the large-scale features.

[0118] The guiding link from large-scale global features to small-scale features is that the large-scale features can provide spatial positioning references for small-scale features.

[0119] The cross-spectral feature collaborative link is a feature that transmits both the nut edge of the visible light channel and the thread texture of the near-infrared channel, and the two features are highly matched in spatial position, belonging to the same bolt.

[0120] Through this filtering process, BiFPN reduces invalid computations while enhancing feature interactions that are crucial for bolt detection, providing high-quality fused features for accurate detection in the subsequent Head network.

[0121] In addition, the anchor for small target detection is set to [5,6,8,10,12,16]. This is a set of target candidate box size parameters preset in the enhanced YOLOv8 detection network to improve the detection and matching accuracy of small bolts, given their extremely small size in images.

[0122] In object detection networks, anchors are a predefined series of rectangles with fixed aspect ratios, used to match the actual size of objects in the image. The network calculates the overlap between the object and the anchors to determine the possible size range of the object, thereby generating more accurate predicted bounding boxes.

[0123] For industrial bolt inspection, small bolts often occupy only 10-50 pixels in an image. The general anchor parameters of traditional YOLOv8 are mostly adapted to medium or large-sized targets, which have a low matching degree with the actual size of small bolts, easily leading to missed detections or misjudgments. Therefore, dedicated anchor parameters are needed for adaptation.

[0124] The specific meanings and applicable scenarios of [5,6,8,10,12,16]:

[0125] This set of parameters consists of three pairs of width and height combinations, with each pair representing the width and height of the anchor, precisely covering the common size range of small bolts in images:

[0126] 5×6: Adapts to local features of the smallest bolt size, such as local areas of small bolt nuts or severely obscured bolt segments, occupying only a tiny area of ​​5-6 pixels in the image, specifically for M6 and M7 bolts;

[0127] 8×10: This type of bolt is suitable for medium to small bolt sizes. The bolt and nut will appear as a hexagonal outline of 8-10 pixels in the image. The thread texture will begin to have recognizable local features. Specifically, this applies to M8 and M9 bolts.

[0128] 12×16: Suitable for larger sizes of small bolts. The overall outline of the nut occupies 12-16 pixels in the image, and the thread area can present a continuous spiral texture. Specifically, it is suitable for M10 and M11 bolts.

[0129] These parameters are set based on actual imaging data of small bolts in industrial scenarios, ensuring that the anchor size closely matches the actual pixel range of the small bolts.

[0130] The operating logic of the anchor settings for small target detection:

[0131] When the enhanced YOLOv8 detection network processes images, for small-scale feature regions, the small bolt details enhanced by CBAM and the local features fused by BiFPN will preferentially use the anchor group [5,6,8,10,12,16] for matching:

[0132] The network compares small-scale feature regions with the width and height of each anchor to calculate the overlap.

[0133] Select the anchor with the highest overlap as the reference to generate the initial prediction box;

[0134] By combining core features such as the edge of the nut and the texture of the thread, the prediction box is fine-tuned, and the precise location and category of the small bolt are finally output.

[0135] Furthermore, the large-scale anchor is set to [20,25,30,35,40,50];

[0136] 20×25: Suitable for medium to large bolts, specifically M12-M14 bolts. The bolt nuts of this type appear as a hexagonal outline of 20-25 pixels in the image, and the thread texture of the bolt part is shown as a continuous spiral grayscale change in the image, which can fully reflect the overall shape of the bolt and its local positional relationship with the surrounding workpiece.

[0137] 30×35: Suitable for larger bolt sizes, specifically M16-M18 bolts. The nut occupies 30-35 pixels in the image, with a clear overall outline and distinct boundaries with adjacent components. The underlying structural features of the bolt can be clearly presented in the near-infrared channel, making it suitable for metal connection scenarios in heavy machinery.

[0138] 40×50: Suitable for large-size bolts, specifically M20 bolts, which occupy more than 40-50 pixels in the image. The geometry of the nut and the overall orientation of the bolt are significant, which can effectively correlate their relative position with the assembly surface. Suitable for the detection of large bolts in outdoor scenarios such as power towers.

[0139] After detecting small targets, large-scale anchors are used for further detection.

[0140] The Enhanced YOLOv8 Detection Network is a target detection model optimized for the features of power grid bolts. By embedding a CBAM convolutional attention module into the Neck network, it strengthens core features such as nut edges and thread textures from both channel and spatial dimensions, while weakening interference from oil stains and debris. It employs a BiFPN feature fusion structure to construct a bidirectional transmission path for the features purified by CBAM, fusing small-scale detailed features with large-scale global features, pruning redundant links and retaining key fusion links to adapt to multiple bolt sizes. Furthermore, it sets dedicated anchor parameters [5, 6, 8, 10, 12, 16] for small target detection, with three pairs of width-to-height combinations adapting to small-size bolts of M6-M7, M8-M9, and M10-M11 respectively, improving detection and matching accuracy. The Enhanced YOLOv8 Detection Network, combined with the Backbone network to extract basic features and the Head network to output target location and category, significantly improves the recognition capability of bolts in small-size, highly interference-prone industrial scenarios, especially enhancing the detection sensitivity and accuracy for small-size bolts.

[0141] After the enhanced YOLOv8 network performs fusion detection on the industrial bolt features in the multispectral image data, the output includes the target location, category, and confidence level.

[0142] Among these requirements, the selection of effective bolt target data must meet two conditions:

[0143] The core features of the target include: the edge data of the nut and the texture data of the thread are clearly distinguishable in the multispectral image, and the feature signal intensity is higher than the set threshold after being enhanced by the CBAM convolutional attention module. Specifically, the contour integrity score of the nut edge is ≥0.8 and the continuity score of the thread texture is ≥0.7.

[0144] The contour integrity score of the nut edge is obtained by calculating the weighted average of the proportion of overlapping edge lengths of the candidate contour and the standard nut contour template and the proportion of effective corners; the continuity score of the thread texture is the average of the proportion of continuous segment lengths that conform to the theoretical period in the actual thread texture and the proportion of thread textures without breaks or missing parts.

[0145] The target has a matching overlap of ≥0.5 with the small target detection anchor or the large-scale anchor, and the confidence level of the predicted category as industrial bolt is ≥0.9.

[0146] The overlap between the target and the detected anchor is calculated by dividing the area of ​​the overlapping region between the target candidate box and the anchor by the difference between the total area of ​​the two and the area of ​​the overlapping region; the confidence score is calculated by the probability value of the target belonging to the industrial bolt category output by the Head network.

[0147] Targets meeting the above conditions will be marked as valid bolt targets, and their two-dimensional coordinates (x, y) in the image, their specification range (e.g., M6-M11 or M12-M20), and multispectral characteristic parameters will be recorded, including:

[0148] The color mean of the visible light channel and the structural entropy value of the near-infrared channel.

[0149] To eliminate invalid interfering targets, the network achieves accurate differentiation through feature comparison:

[0150] For gaskets, those with regular annular contours but no thread texture data and no underlying helical structure features in the near-infrared channel, and whose feature difference from industrial bolts is ≥0.6, will be directly marked as interference targets and eliminated. The feature difference is calculated by measuring the degree of mismatch between the gasket's regular annular contours, lack of thread texture, and lack of underlying helical structure in the near-infrared channel and the core features of industrial bolts, and thus obtaining the difference value between the two.

[0151] For debris, characterized by irregular shape, lack of complete nuts or thread structure, a score ≤0.3 in the spatial attention weight map, and a matching overlap of <0.2 with any industrial bolt anchor, it will also be judged as an invalid interference target.

[0152] The score of debris in the spatial attention weight map: Based on the spatial attention mechanism, the weight value obtained by the debris region in the feature map is calculated. This value is determined by its matching degree with the geometric features of the bolt. The matching overlap between debris and industrial bolt anchor is obtained by calculating the ratio of the intersection area of ​​the debris candidate box and all industrial bolt anchors to the union area of ​​the two.

[0153] Ultimately, the effective bolt target data output in step S1 includes both accurate image location information after multispectral feature fusion and specification-related features of the industrial bolts. This provides reliable target reference data for the 6DoF attitude calculation of the industrial bolts and the robotic arm grasping path planning in the subsequent step S2. At the same time, by eliminating invalid interference targets such as gaskets and debris, the processing efficiency and accuracy of subsequent processes are ensured.

[0154] The specific effective bolt target data for S1 includes:

[0155] Image location information: the two-dimensional coordinates (x, y) of the industrial bolt in the multispectral image; x is the horizontal coordinate and y is the vertical coordinate.

[0156] The specification range identifier indicates the range of bolt specifications to which it belongs, i.e., M6-M11 (small size) or M12-M20 (large size), which is used for subsequent adaptation and grabbing strategies.

[0157] The core feature data includes: nut edge data and thread texture data; the nut edge data is the nut edge contour information enhanced by the CBAM module, including: edge integrity parameters, angle and position coordinates of the hexagonal head corner; the thread texture data is the thread structure information extracted by the near-infrared channel, including: thread pitch, helix direction and continuity parameters;

[0158] Multispectral feature parameters include: visible light channel features and near-infrared channel features; visible light channel features include: color mean and texture complexity of the bolt area; near-infrared channel features include: grayscale distribution entropy of the underlying structure and structural integrity parameters after penetration interference;

[0159] The confidence data includes: category confidence and matching overlap. Category confidence is the probability value of the industrial bolt category output by the Head network. Matching overlap is the intersection-union ratio with the corresponding anchor. The confidence data is used to filter valid targets.

[0160] Based on the valid bolt target data output from step S1, S2 completes the attitude calculation and motion compensation of the industrial bolt, providing a three-dimensional spatial positioning reference for the robotic arm's grasping:

[0161] Image feature point data extracted from effective bolt target data is used as a key geometric reference for attitude calculation;

[0162] The nut feature points are based on the hexagonal head vertex coordinates extracted from the nut edge data, including: the three-dimensional coordinates (x1, y1, z1) to (x6, y6, z6) of the 6 vertices in the multispectral image, and the three-dimensional coordinates of the midpoints of the edges (the (x, y, z) values ​​of the midpoints of the 6 edges).

[0163] The x and y coordinates are derived from the pixel location of the multispectral image, while the z coordinate is obtained by synchronous acquisition from the depth sensor or by geometric deduction based on the standard thickness of the M6-M20 nut. The synchronous acquisition from the depth sensor reflects the distance of the vertex relative to the sensor. These points together reflect the symmetry and spatial orientation of the nut.

[0164] The thread feature points are based on the three-dimensional coordinates of the thread start point, end point, and helical pattern inflection point extracted from the thread texture data; the x and y coordinates are obtained from the near-infrared channel image, and the z coordinate is obtained through a depth sensor or calculated based on the thread helix angle and (x,y) offset, which are used to reflect the helical direction and length of the screw.

[0165] Feature point association: Record the spatial distance between each three-dimensional feature point to form a topological structure containing three dimensions: x, y, and z, ensuring the spatial consistency of the feature point set.

[0166] 6DoF attitude refers to the position and attitude angle of the bolt in three-dimensional space. The calculation process is as follows:

[0167] The extracted 3D feature point set is matched with a pre-set 3D model feature library of bolts of the same specification, and the initial pose of the bolt in the camera coordinate system is calculated by the PnP algorithm.

[0168] By combining multispectral feature parameters, including the color mean and texture complexity of the visible light channel, the grayscale distribution entropy value and structural integrity parameters of the near-infrared channel, the LM algorithm is used to iteratively optimize the initial pose, reduce the error caused by feature point occlusion, and control the position accuracy within ±0.1mm and the pose angle accuracy within ±0.5°.

[0169] Specifically, using the initial bolt posture output by the PnP algorithm as the starting point for iteration, the color mean and texture complexity of the visible light channel, the grayscale distribution entropy of the near-infrared channel, and the structural integrity parameters are transformed into feature constraints. In each iteration, the error between the measured features at the current posture and the features in the feature library of the same specification bolt 3D model is first calculated.

[0170] The weights of each feature are dynamically adjusted according to the magnitude of the error. This process is repeated until the position deviation is less than 0.1 mm and the attitude angle deviation is less than 0.5°, and finally the optimized high-precision attitude data is output.

[0171] The pre-defined 3D model feature library for bolts of the same specification is a structured feature set built based on standard parameters and measured data of industrial bolts, providing a unified reference benchmark for attitude calculation. Its generation process is as follows:

[0172] Based on the dimensional parameters of bolts of various specifications from M6 to M20 in the international standard ISO898, a standard 3D model was generated using the 3D modeling software SolidWorks.

[0173] In a standard 3D model, feature points used for pose matching are calibrated manually or automatically, including:

[0174] The three-dimensional coordinates of the 6 vertices and the midpoints of the 6 sides of the nut;

[0175] Coordinates of the thread start point, end point, and turning point of each thread turn;

[0176] The correlation parameters between feature points include: the distance between adjacent vertices and the axial distance corresponding to the thread pitch.

[0177] This feature library is stored in the form of a database and is classified according to bolt size (M6-M20) and head shape (hexagonal head, round head). It can be called in real time by the PnP algorithm to provide a high-precision reference template for attitude calculation.

[0178] The PnP algorithm is used to calculate the initial pose of the bolt in the camera coordinate system by using the correspondence between the three-dimensional feature points of the bolt and the feature points of the model to solve the spatial transformation matrix from the model coordinate system to the camera coordinate system.

[0179] In step S2, the 6DoF attitude calculation based on the PnP algorithm includes:

[0180] The input is the extracted set of 3D feature points of industrial bolts, including: the 3D coordinates of the nut and thread, as well as the standard 3D coordinates from the preset 3D model feature library of bolts of the same specification.

[0181] Among them, the coordinates of the three-dimensional feature points extracted from the effective bolt target data in the camera coordinate system are {P1, P2, ..., P...} n The coordinates of the standard feature points in the bolt 3D model feature library in the model coordinate system are {Q1, Q2, ..., Q}. n}, where n≥6, and n is the number of feature points;

[0182] The 3D coordinates of the i-th 3D feature point extracted from the effective bolt target data in the camera coordinate system are: These are the coordinates of the point on the X, Y, and Z axes of the camera coordinate system, where c represents the camera coordinate system.

[0183] The three-dimensional coordinates of the i-th standard feature point in the bolt 3D model feature library in the model coordinate system are: These are the coordinates of the point on the X, Y, and Z axes of the model coordinate system, respectively; m represents the model coordinate system.

[0184] And P i and Q i A one-to-one correspondence has been established through feature matching, indicating that they are the same physical feature points. The spatial distribution of these feature points directly reflects the attitude characteristics of the bolt.

[0185] The input 3D feature points are preprocessed to remove outliers caused by noise, occlusion, etc., to ensure that the feature points used in the calculation are accurate and reliable.

[0186] By calculating the Euclidean distance between each point in the 3D feature point set and the standard points in the model feature library, the closest point is found as the matching point, and a one-to-one correspondence is established. At the same time, the spatial distance information in the feature point association relationship is combined for verification, and incorrect matching pairs are eliminated to ensure the accuracy of matching.

[0187] Using the established feature point correspondence, the rotation matrix R of the bolt in the camera coordinate system is solved based on the PnP algorithm. m→c Translation vector t m→c .

[0188] Among them, the spatial transformation of the bolt satisfies Pi =R m→c ·Q i +t m→c Rotation matrix R m→c The bolt's pose is described by a 3×3 rotation matrix, representing the rotation of the bolt model coordinate system m relative to the camera coordinate system c; the translation vector t... m→c The bolt's position is described by a 3×1 translation vector, and the bolt model coordinate system origin m is described by the position of the bolt model origin in the camera coordinate system c. These two parameters determine the bolt's initial 6DoF pose.

[0189] Based on rotation matrix R m→c The Euler angle decomposition algorithm is used to convert it into three attitude angles (roll). c pitch c ,yaw c ), where c represents the camera coordinate system, roll, pitch, and yaw are the roll angle, pitch angle, and yaw angle, respectively; represent the rotation angles around the X-axis of the camera coordinate system, used to reflect the torsional state of the bolt around its own axis; the rotation angles around the Y-axis of the camera coordinate system, used to reflect the tilt of the bolt in the vertical direction; and the rotation angles around the Z-axis of the camera coordinate system, used to reflect the deflection state of the bolt in the horizontal direction.

[0190] To eliminate the interference of robotic arm vibration on positioning, motion compensation is achieved by integrating IMU data. The inertial measurement unit (IMU) collects the vibration acceleration and angular velocity of the robotic arm end in real time at a frequency of 1-1.5kHz, reflecting the vibration state of the robotic arm in three-dimensional space.

[0191] The IMU data and the initial attitude data of the bolt are synchronized in time and space. The position and angle offsets caused by vibration are calculated by Kalman filtering, and the 6DoF attitude is corrected in real time. The angle offset is removed from the initial attitude angle to obtain the compensated attitude angle in the camera coordinate system. After motion compensation, the 6DoF attitude in the camera coordinate system is transformed to the robot coordinate system through a preset camera-robot coordinate transformation matrix. The final output is the three-dimensional coordinate data and attitude angle data in the robot coordinate system.

[0192] Among them, the preset camera-robotic arm coordinate transformation matrix is ​​a rigid transformation matrix used to transform the three-dimensional coordinates and attitude data in the camera coordinate system to the robotic arm coordinate system;

[0193] This matrix is ​​a 4×4 homogeneous transformation matrix. R c→r t is a rotation matrix representing the rotation relationship from the camera coordinate system c to the robot arm coordinate system r. It is used to transform points in the camera coordinate system to the robot arm coordinate system, reflecting the installation angle relationship between the camera and the robot arm; c→rThe translation vector describes the position of the camera coordinate system origin in the robot arm coordinate system, and is used to determine the installation position of the camera in the robot arm's workspace.

[0194] The camera-robotic arm coordinate transformation matrix was obtained in advance through a hand-eye calibration experiment. The specific process is as follows:

[0195] Fix the high-precision checkerboard calibration plate onto the end effector of the robotic arm to ensure that the coordinate system of the calibration plate coincides with the coordinate system of the end effector of the robotic arm.

[0196] The robotic arm is controlled to move the calibration board in at least 10 different poses within the camera's field of view, and the following are recorded simultaneously for each pose: the calibration board image captured by the camera and the end effector pose fed back by the robotic arm controller.

[0197] Using the Tsai-Lenz or AX=XB hand-eye calibration algorithm, iterative calculations are performed on multiple sets of pose data to solve for the transformation matrix between the camera and the robotic arm.

[0198] Finally, S2 outputs the three-dimensional coordinates and attitude angle data of the industrial bolt in the robot arm coordinate system r.

[0199] Among them, the three-dimensional position coordinates are the result after deducting the position offset caused by vibration from the initial calculated position; the attitude angle is the result after eliminating the angle offset caused by vibration from the initial attitude angle, which more realistically reflects the tilt and orientation of the bolt in space.

[0200] S3 uses the three-dimensional coordinates (X, Y) of the industrial bolt in the robotic arm coordinate system output by S2. r ,Y r Z r and attitude angle (roll) r pitch r ,yaw r Based on this, a coordinate transformation and normal vector compensation algorithm is used to generate the gripping target position that the robotic arm can execute, thus solving the gripping deviation problem caused by the tilt of the assembly surface; where r is the robotic arm coordinate system, and roll, pitch, and yaw are the roll angle, pitch angle, and yaw angle, respectively.

[0201] The motion control of the robotic arm depends on the angle commands in the joint space or the path points in the Cartesian space. The three-dimensional coordinate data output in step S2 needs to be converted into a motion coordinate format that the robotic arm controller can recognize.

[0202] The conversion process includes:

[0203] The rectangular coordinates in the robotic arm coordinate system are directly mapped to the target position parameters in Cartesian space, with the unit uniformly set to millimeters and three decimal places retained.

[0204] The attitude angles are converted into rotation commands for the robotic arm's end effector, using Euler angles to ensure consistency with the attitude parameter protocol of the robotic arm controller; the converted motion coordinates must meet the robotic arm's motion range constraints, and a path replanning prompt is triggered when the range is exceeded.

[0205] After the transformation is complete, output the motion coordinates P. ini =(X f ,Y f Z f ,roll f pitch f ,yaw f ), where f represents the coordinate transformation after motion.

[0206] Since the inclination of the assembly surface where the industrial bolt is located directly affects the gripping stability, its spatial orientation needs to be described by the normal vector.

[0207] Point cloud data of the assembly surface is collected using a depth sensor, and the normal vector is calculated using a plane fitting algorithm. n x ,n y ,n z These are the components of the normal vector along the X, Y, and Z axes of the robot arm coordinate system. The normal vector must be consistent with the robot arm coordinate system. When obtained through visual measurement, it needs to be transformed using the camera-robot arm coordinate transformation matrix t. c→r Switch to the robot arm coordinate system.

[0208] To ensure that the contact direction between the robotic arm's end effector, the gripper, and the bolt is perpendicular to the assembly surface, a compensation algorithm is used to correct the motion coordinates. λ is the compensation coefficient, dynamically adjusted according to bolt specifications to ensure a safe clearance between the clamps and the mounting surface; P target This is the corrected target location.

[0209] After correction by the compensation algorithm, the final output of step S3 includes: the corrected three-dimensional coordinates in the robot arm coordinate system, the attitude angle adapted to the robot arm controller, the assembly surface normal vector parameters and the compensation coefficient, which serve as auxiliary information for path planning in step S4.

[0210] Step S4 uses the corrected target position (X) output from step S3 to capture the target position. tar ,Y tar Z tar ), attitude angle (roll) tar pitch tar ,yaw tar and assembly surface normal vector Using the bolt specifications as input, a collision-free motion trajectory is generated through a path planning algorithm, and combined with a dynamic matching grasping strategy, a complete execution solution is provided for the robotic arm to grasp precisely.

[0211] The core objective of path planning is to generate the optimal trajectory from the current position to the grasping point under the kinematic constraints of the robotic arm, ensuring smooth, collision-free, and efficient movement.

[0212] Starting point: The joint angle or Cartesian coordinate of the current position of the robotic arm; the starting point is s, and the starting point coordinates are P. s =(X s ,Y s Z s ,roll s pitch s ,yaw s It is necessary to ensure that the endpoint orientation is parallel to the bolt and nut plane, through the assembly surface normal vector. calibration.

[0213] Endpoint: The corrected target position and attitude angle output from step S3, denoted as...

[0214] P target =(X tar ,Y tar Z tar ,roll tar pitch tar ,yaw tar )

[0215] Collect point cloud data of the environment around the bolt, construct a three-dimensional obstacle model, set the dynamic obstacle avoidance detection radius to 150mm, and mark the prohibited areas within this range.

[0216] The path is planned using a fifth-order polynomial interpolation algorithm, including:

[0217] First, a smooth path from the starting point to the ending point is searched using a fifth-order polynomial interpolation algorithm while satisfying the constraints, thus avoiding obstacles.

[0218] Simultaneously, velocity planning is performed, with the maximum acceleration set at 2 m / s². 2 When the end of the robotic arm is 50mm away from the target position, a deceleration mechanism is triggered to ensure smooth movement when approaching the target.

[0219] Among them, the fifth-order polynomial interpolation algorithm constructs a fifth-order polynomial function, s(t) = a5t. 5 +a4t 4 +a3t 3 +a2t 2+a1t+a0 describes the positional change of the robotic arm's end effector in the time dimension, where t is the motion time and a0-a5 are polynomial coefficients;

[0220] Input boundary conditions, constrained by parameters of the start and end points, including: position coordinates, velocity, and acceleration of the start and end points;

[0221] The unique quintic polynomial function is determined by solving the simultaneous boundary condition equations.

[0222] Based on the solved polynomial function, the position and orientation of the robotic arm's end effector are calculated at fixed time intervals during the motion time from the starting point to the ending point, generating a series of continuous intermediate nodes.

[0223] When a segment of the trajectory enters a prohibited area, the path is corrected by fine-tuning the polynomial coefficients to ensure that the trajectory always remains in a safe area.

[0224] The final output is the movement path of the robotic arm.

[0225] The planned movement path is represented by a series of intermediate nodes, with the node spacing dynamically adjusted according to the bolt specifications. Specifically:

[0226] Small bolts (M6-M11): node spacing ≤ 5mm to ensure movement accuracy when close to the gripping point;

[0227] Large bolts (M12-M20): node spacing ≤10mm, balancing efficiency and stability.

[0228] Each node contains three-dimensional coordinates and attitude angles in the robotic arm coordinate system, forming continuous trajectory data, which serves as the movement path of the robotic arm from its current position to the industrial bolt gripping point.

[0229] The gripping strategy needs to be dynamically adjusted based on bolt sizes M6-M20, attitude angle, and assembly surface environment. Key parameters include jaw opening and gripping force.

[0230] The initial opening amplitude, D, is determined based on the diagonal dimension of the bolt and nut. open =D nut +2δ, where D nut δ represents the standard diagonal dimension of the nut, and δ is the safety margin.

[0231] Small-sized bolts (M6-M11): Use a clamping force of 0.5-1.5N, of which 1.0-1.5N is used for carbon steel and 0.5-1.0N is used for stainless steel;

[0232] Large bolts (M12-M20): Use clamping force of 2.0-4.0N, increasing with size.

[0233] At the same time, the clamping force needs to be adjusted in conjunction with the attitude angle.

[0234] The generated crawling strategy needs to be verified through simulation:

[0235] Simulate the jaw closing process in a virtual environment to check whether the contact points between the jaws and the nut are evenly distributed;

[0236] When contact offset occurs, the gripper attitude angle is automatically fine-tuned within ±1°, and the opening amplitude and clamping force are recalculated.

[0237] The final output data of step S4 includes: path data and crawling strategy data.

[0238] In S5, the robotic arm is controlled by outputting path data and grasping strategy data in step S4. Based on the contact force data fed back by the six-dimensional force sensor and the torque data fed back by the strain gauge, the robotic arm dynamically adjusts the clamping force and rotation torque to achieve precise grasping and posture calibration of industrial bolts.

[0239] The path data and grasping strategy data are transmitted to the robotic arm controller through the CANopen communication module. The CANopen communication module adopts a dual-channel redundant design with a baud rate of 1Mbps, a synchronization period of 1ms, and a maximum of 32 nodes.

[0240] Specifically, the CANopen communication module is designed based on the open-source communication standard of the CAN bus protocol. It is adapted to the needs of multi-device collaborative control in industrial automation scenarios. It can convert the path data and grasping strategy data output in step S4 into an instruction format that the robotic arm controller can recognize, and realize real-time bidirectional data transmission to ensure that the robotic arm moves according to the planned trajectory.

[0241] The dual-channel redundancy design includes:

[0242] At the physical level, the CANopen communication module has two independent CAN bus channels built in. Each channel is equipped with an independent transceiver, isolation circuit and signal verification unit. The two channels work in parallel and are completely electrically isolated.

[0243] At the data transmission level, the same control command, including path node coordinates and clamping force parameters, is sent to the robotic arm controller simultaneously through two independent CAN bus channels.

[0244] The robotic arm controller has a built-in dual-channel data comparison mechanism. When the data from the two channels are consistent, the control command is executed. When there is a discrepancy, the channel with the higher signal strength and the correct checksum is automatically selected, and the self-diagnosis of the faulty channel is triggered.

[0245] After receiving and parsing path data and grasping strategy data through the CANopen communication module, the robotic arm controller generates motion commands for each joint, which are then executed by the servo drive array.

[0246] The servo drive array is based on a fully closed-loop control architecture. After receiving joint motion commands, it adjusts the position loop, speed loop, and current loop in three levels to make the end effector of the robotic arm move along the planned trajectory. The position resolution of the servo drive array reaches 0.001mm, ensuring that the positioning error of each intermediate node is ≤0.05mm, which meets the high precision requirements of bolt gripping.

[0247] During the movement, the servo drive array provides real-time feedback on the actual position and speed of each joint, compares it with the command value, and dynamically corrects the output through the PID algorithm to ensure that the end effector of the robotic arm strictly follows the trajectory nodes in the path data.

[0248] When the intelligent gripper at the end of the robotic arm grasps, it uses an integrated six-dimensional force sensor to detect the three-dimensional force and three-dimensional torque during the contact process between the gripper and the bolt. The six-dimensional force sensor of the intelligent gripper integrates a high-precision strain gauge with a sensitivity coefficient of 2.0-2.2, a measurement range of 0-5 N·m, and a sampling frequency of 1 kHz. It is used to monitor the rotational torque generated by the contact of the nut when the gripper closes in real time.

[0249] When the robotic arm reaches the gripping point and drives the gripper to close, the six-dimensional force sensor and strain gauges are activated simultaneously to collect force and torque signals in real time. Based on this data, the robotic arm controller forms a dual closed-loop regulation through an adaptive gripping force control algorithm and a real-time torque control algorithm.

[0250] The three-dimensional force refers to the force along the X, Y, and Z axes of the robot arm coordinate system when the gripper contacts the bolt, and the three-dimensional torque refers to the torque around the X, Y, and Z axes of the robot arm coordinate system when the gripper contacts the bolt. It shares the same spatial reference with the trajectory node coordinates of the path planning in step S4.

[0251] When the robotic arm reaches the bolt gripping point according to the planned path and drives the gripper to begin closing, the six-dimensional force sensor is activated simultaneously to collect and preprocess the force signal generated by the contact between the gripper and the bolt / nut in real time.

[0252] The robotic arm controller dynamically adjusts the gripping force based on contact force data using a preset adaptive gripping force control algorithm. The preset adaptive gripping force control algorithm is F. grip =K(1+e -μ·d ), where F grip To represent the actual gripping force of the robotic arm's intelligent gripper, K is the base coefficient, which is the initial gripping force parameter based on the bolt specifications and originates from the gripping strategy data generated by S4. d is the deviation coefficient, which represents the deviation between the current actual gripping force and the target reference force. μ is the adjustment coefficient, which is used to control the smoothness of the gripping force adjustment.

[0253] Dynamic adjustment logic:

[0254] In the initial contact phase between the gripper and the bolt, when the gripper has just made contact with the bolt, the force along the Z-axis of the robotic arm coordinate system, i.e. the normal pressure perpendicular to the assembly surface, is relatively small. The algorithm increases the clamping force by increasing the coefficient of K, driving the gripper to close and make tight contact with the nut, thus preventing the bolt from sliding inside the gripper.

[0255] As the grippers remain closed, the normal pressure value perpendicular to the assembly surface gradually approaches K. The algorithm then reduces the adjustment range to bring the clamping force closer to the target value.

[0256] When the normal force perpendicular to the assembly surface exceeds the target force, the algorithm triggers attenuation adjustment, slightly loosening the grippers to reduce the clamping force. Furthermore, for easily deformable thin-walled nuts, an additional upper limit threshold is set to prevent the nut from being damaged by clamping.

[0257] The real-time torque control algorithm takes the target torque of the bolt specification in the grasping strategy data as the core benchmark, and forms a collaborative closed loop with the adaptive clamping force control algorithm to achieve dynamic balance between torque and clamping force.

[0258] The target torque is set according to bolt size: 0.3-0.8 N·m for small bolts and 1.0-2.0 N·m for large bolts.

[0259] The deviation from the target torque is calculated based on the real-time collected rotational torque value, and adjustments are made according to the following logic:

[0260] When the strain gauge detects that the actual torque is lower than the target value and the deviation exceeds the threshold, the rotational torque is increased by linearly increasing the current output of the gripper drive motor. The current adjustment is linked to the Z-axis positive pressure of the six-dimensional force sensor to ensure that the torque is maintained within the target proportional range during the torque increase process.

[0261] When the actual torque exceeds the target value, the motor current is immediately reduced, and the torque change in the X / Y axis direction is monitored by a six-dimensional force sensor. When the nut is detected to be tilting due to over-torque, the gripper attitude angle is adjusted synchronously until the torque deviation returns to within ±0.05 N·m.

[0262] The adjusted clamping force and torque commands are transmitted in real time to the servo drive unit of the gripper via the CANopen communication module, driving the tiny displacement of the gripper fingers to achieve precise force adjustment.

[0263] Exemplary system:

[0264] A method and system for intelligent identification and precise gripping control of industrial bolts based on depth vision, including:

[0265] The visual perception subsystem is used for intelligent identification, feature extraction, and 3D pose calculation of bolts.

[0266] The visual perception subsystem achieves accurate bolt location and classification through an enhanced YOLOv8 detection network. It is optimized to address interference factors such as oil stains and partial occlusion of bolts in industrial scenarios, and strengthens key features such as the nut edge and thread texture of the bolt. At the same time, it effectively integrates image information of different scales, thereby accurately locating the bolt position and determining its specifications in complex environments, providing a reliable target area for subsequent feature extraction and attitude calculation.

[0267] The visual perception subsystem provides high-quality image input to the enhanced YOLOv8 detection network through a multispectral imaging module. The multispectral imaging module performs imaging through two spectral channels: visible light and near-infrared. The visible light channel can clearly capture the color and surface texture details of the bolt, while the near-infrared channel can penetrate interference such as oil stains and rust to present the true shape of the bolt. At the same time, with adjustable polarized illumination, the impact of metal reflection on image quality is reduced, ensuring that the acquired bolt images can meet the feature extraction requirements of the detection network.

[0268] The visual perception subsystem calculates the three-dimensional spatial attitude of the bolt through the three-dimensional attitude calculation module, uses the PnP algorithm to solve the six-degree-of-freedom attitude parameters of the bolt, and integrates motion data collected by the inertial measurement unit for compensation to eliminate errors caused by environmental vibration and other factors. Finally, it outputs the three-dimensional coordinates and attitude angles in the robotic arm coordinate system, providing direct position and attitude basis for the robotic arm's precise grasping.

[0269] Specifically, the image data acquired by the multispectral imaging module is input into the enhanced YOLOv8 detection network. After processing, the bolt positioning and specification information output is input into the three-dimensional attitude calculation module, which finally generates three-dimensional attitude data in the robot arm coordinate system.

[0270] The intelligent decision-making subsystem is used to convert visual perception data into action commands that can be executed by the robotic arm.

[0271] The intelligent decision-making subsystem converts bolt image coordinates to robotic arm operating coordinates through a spatial coordinate mapping module. It establishes a connection between the visual coordinate system and the robotic arm base coordinate system based on the hand-eye calibration matrix, and performs deviation correction by combining the normal vector parameters of the assembly surface. This ensures the accuracy of bolt position mapping in the robotic arm's operating space and provides an accurate spatial coordinate reference for subsequent trajectory planning.

[0272] The intelligent decision-making subsystem generates the optimal motion path for the robotic arm through the motion trajectory planning module. Based on the three-dimensional posture parameters of the bolt and the grasping operation requirements, it uses a fifth-order polynomial interpolation algorithm to plan a smooth trajectory, sets reasonable acceleration and speed thresholds, and integrates real-time obstacle avoidance logic to dynamically detect obstacles in the work space and adjust the path, ensuring the safety and efficiency of the robotic arm's movement.

[0273] The intelligent decision-making subsystem determines the appropriate clamping scheme through the grasping strategy optimization module. Based on the bolt specifications and models output by the enhanced YOLOv8 detection network, it matches the preset clamping force parameters and jaw posture. Combined with the bolt tilt angle provided by the three-dimensional posture calculation module, it dynamically adjusts the opening and closing range of the jaws and the contact point position to ensure stable grasping of bolts of different specifications.

[0274] Specifically, the three-dimensional posture data output by the visual perception subsystem is first fed into the spatial coordinate mapping module to complete the coordinate transformation. The transformed robotic arm operation coordinates are then input into the motion trajectory planning module and the grasping strategy optimization module to generate motion path and gripping scheme instructions.

[0275] The precision execution subsystem is used to perform the physical gripping operation of the bolt;

[0276] The precision execution subsystem controls the movement of the robotic arm through the servo drive module, receives trajectory instructions output by the intelligent decision-making subsystem, drives the motors of each joint in a fully closed-loop control mode, provides real-time feedback on position and speed information and makes dynamic adjustments to ensure that the end effector of the robotic arm moves precisely along the planned trajectory, thereby achieving the positioning of the bolt.

[0277] The precision execution subsystem completes the bolt gripping action through the intelligent gripper module. It integrates a six-dimensional force sensor to monitor the force feedback data in real time during the gripping process. Based on the parameters set by the gripping strategy optimization module, it automatically adjusts the gripping force to avoid bolt deformation or slippage while ensuring the stability of the gripping posture.

[0278] The motion path and clamping scheme instructions generated by the intelligent decision-making subsystem are transmitted to the servo drive module and the intelligent gripper module. Combined with the real-time feedback data from the force sensor, closed-loop control is completed to achieve precise bolt gripping.

[0279] Based on the preferred embodiments of the present invention described above, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for intelligent identification and precise grasping control of industrial bolts based on depth vision, characterized in that, Including the following steps: S1. Acquire multispectral image data of industrial bolts, and use an enhanced YOLOv8 network to fuse and detect the feature data of industrial bolts in the multispectral images, filter out valid bolts, and output target data; among which, the feature data of industrial bolts include nut edge data and thread texture data; S2. Extract image feature point data from the target data of the effective bolts, calculate the 6DoF attitude of the bolts in the camera coordinate system, perform motion compensation, and convert the attitude data to the robot arm coordinate system through the hand-eye calibration matrix. Output the three-dimensional coordinates and attitude angle data of the industrial bolts in the robot arm coordinate system. S3. Convert the three-dimensional coordinate data in the robotic arm coordinate system into motion coordinates that the robotic arm can recognize, and combine them with the normal vector data of the assembly surface to correct the target position for gripping industrial bolts. S4. Plan the motion path of the robotic arm from its current position to the industrial bolt gripping point, and generate gripping strategy data based on the bolt specifications; the gripping strategy data includes gripper opening amplitude data and gripping force parameter data. S5. The robotic arm is controlled by path data and grasping strategy data. Based on the contact force data fed back by the six-dimensional force sensor, the clamping force and torque are dynamically adjusted to accurately grasp industrial bolts.

2. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S1, the enhanced YOLOv8 network is improved based on the YOLOv8 network, including: embedding the CBAM convolutional attention module into the Neck network to enhance the edge and thread texture features of the nut from the channel and spatial dimensions; using the BiFPN feature fusion structure to construct a bidirectional transmission path, fusing small-scale detail features with large-scale global features; and setting the anchor parameters for small target detection to 5×6, 8×10, and 12×16, respectively adapted to M6-M7, M8-M9, and M10-M11 bolts.

3. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S1, the multispectral image data includes visible light and near-infrared dual-channel images; the visible light channel has a wavelength of 400-700nm, and the near-infrared channel has a wavelength of 850nm; the illumination angle is dynamically adjusted from 0° to 180° by polarizing the illumination direction, and the polarization direction is automatically rotated to be perpendicular to the reflection direction when specular reflection on the bolt surface is detected; the visible light channel is used to capture the bolt color and surface texture features, and the near-infrared channel is used to penetrate oil and rust interference to collect the underlying structural features of the bolt.

4. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S2, the 6DoF attitude calculation includes: extracting the three-dimensional coordinates of the nut feature points and the three-dimensional coordinates of the thread feature points; matching the feature point set with the preset feature library of three-dimensional bolt models of the same specification; calculating the initial attitude using the PnP algorithm; and iterating using the LM algorithm.

5. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S2, the motion compensation achieved by fusing the robotic arm vibration data includes: using an inertial measurement unit to collect the robotic arm vibration acceleration and angular velocity, calculating the position offset and angular offset through Kalman filtering, and performing real-time correction of the six-degree-of-freedom attitude.

6. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: Correcting the target position using a compensation algorithm based on normal vector data includes: The point cloud data of the assembly surface is collected, and the normal vector of the assembly surface in the robot arm coordinate system is calculated by the plane fitting algorithm. The compensation coefficient is dynamically determined based on the specifications of the industrial bolts. The compensation coefficient is used to control the reserved safety clearance between the clamp and the assembly surface. The initial gripping target position is spatially offset and corrected along the normal vector direction so that the corrected gripping target position satisfies that the contact direction of the robotic arm end effector is perpendicular to the assembly surface.

7. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S4, path planning includes: A fifth-order polynomial interpolation algorithm is used to generate collision-free trajectories, and a maximum acceleration and deceleration mechanism when approaching the target are set. The spacing between path nodes is also dynamically adjusted based on the bolt specifications. A fifth-order polynomial function is constructed to describe the positional change of the robotic arm's end effector in the time dimension. The position coordinates, velocity, and acceleration of the starting and ending points are input as boundary conditions to determine the polynomial function. The position and attitude of the robotic arm's end effector are calculated at fixed time intervals to generate continuous intermediate nodes. When the trajectory enters a prohibited area, the polynomial coefficients are fine-tuned to correct the path. Set the maximum acceleration to 2 m / s². 2 When the end of the robotic arm is 50mm away from the target position, a deceleration mechanism is triggered; the path node spacing is dynamically adjusted based on the bolt specifications, with the node spacing of small bolts (M6-M11) ≤ 5mm and the node spacing of large bolts (M12-M20) ≤ 10mm.

8. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S4, generating the grabbing strategy data adapted to industrial bolt specifications includes: The initial opening width of the jaws is determined based on the standard diagonal dimensions of industrial bolts and nuts. The initial opening width is the sum of the standard diagonal dimensions of the nut and the safety margin. The clamping force parameters are set according to the specifications of industrial bolts. The clamping force for small bolts (M6-M11) is 0.5-1.5N, of which 1.0-1.5N is used for carbon steel bolts and 0.5-1.0N is used for stainless steel bolts. The clamping force for large bolts (M12-M20) is 2.0-4.0N, and increases with the bolt size. By combining the industrial bolt attitude angle to correct the clamping force parameters, the clamping jaw closing process is simulated in a virtual environment to check the distribution of contact points between the clamping jaw and the nut. If there is contact offset, the clamping jaw attitude angle is finely adjusted within ±1° and the opening amplitude and clamping force are recalculated.

9. The method for intelligent identification and precise gripping control of industrial bolts based on depth vision according to claim 1, characterized in that: In step S5, the robotic arm dynamically adjusts the clamping force and torque based on the contact force data fed back by the six-dimensional force sensor, including: The path data and grasping strategy data are transmitted to the robotic arm controller through the CANopen communication module. The CANopen communication module adopts a dual-channel redundant design with a baud rate of 1Mbps and a synchronization period of 1ms. After parsing the data, the robotic arm controller generates motion commands for each joint, which are then executed by the servo drive array. The servo drive array uses three levels of adjustment—position loop, speed loop, and current loop—to make the end effector of the robotic arm move along the planned trajectory. The six-dimensional force sensor and strain gauge of the intelligent gripper at the end of the robotic arm synchronously collect contact force and rotational torque signals. Based on these signals, the robotic arm controller dynamically adjusts the gripping force and rotational torque through an adaptive gripping force control algorithm and a real-time torque control algorithm.

10. A depth vision-based intelligent identification and precise gripping control system for industrial bolts, based on the depth vision-based intelligent identification and precise gripping control method for industrial bolts as described in claims 1-9, characterized in that... It includes a visual perception subsystem, an intelligent decision-making subsystem, and a precise execution subsystem: The visual perception subsystem includes: a multispectral imaging module, an enhanced YOLOv8 detection network module, and a 3D pose calculation module. The multispectral imaging module is used to acquire multispectral image data of the environment where the industrial bolts are located. The enhanced YOLOv8 detection network module is used to fuse and detect the feature data of the industrial bolts in the multispectral images and select valid bolts. The 3D pose calculation module is used to extract image feature point data, solve 6DoF pose, and fuse robotic arm vibration data to achieve motion compensation. The intelligent decision-making subsystem includes: a spatial coordinate mapping module, a motion trajectory planning module, and a grasping strategy optimization module. The spatial coordinate mapping module is used to convert three-dimensional coordinate data into motion coordinates that the robotic arm can recognize and to correct the position of the grasping target by combining normal vector data. The motion trajectory planning module is used to plan the path data of the robotic arm from the current position to the grasping point. The grasping strategy optimization module is used to generate grasping strategy data that is adapted to the specifications of industrial bolts. The precision execution subsystem includes: a CANopen communication module, a servo drive array, and an intelligent gripper module; the CANopen communication module is used to transmit path data and gripping strategy data, the servo drive array is used to execute joint motion commands to make the robotic arm move along the planned trajectory, and the intelligent gripper module is used to dynamically adjust the gripping force and torque based on the contact force data fed back by the six-dimensional force sensor to achieve precise gripping.

Images