Visual feedback taking control method and device suitable for multi-specification hardware

Abstract

The application discloses a visual feedback material taking control method and device suitable for multi-specification hardware, relates to the related technical field of material taking control, and comprises the following steps: collecting real-time images of a hardware stacking area through a multi-light source vision unit, performing multi-specification hardware segmentation and identification based on three-dimensional point cloud data; performing material taking analysis based on specification type information, generating a material taking parameter set; performing real-time visual tracking feedback on material taking path parameters; calculating deviations of the material taking path parameters based on a pose change trend diagram, correcting the material taking trajectory according to the path deviation amount, and performing control optimization in combination with material taking clamp parameters to formulate a material taking control scheme. The application solves the technical problems of insufficient visual detection and identification precision of multi-specification hardware, weak real-time correction capability of dynamic material taking paths, and poor multi-specification hardware sorting precision and operation efficiency in the prior art, and achieves the technical effects of improving the visual detection and identification precision of multi-specification hardware and the sorting precision and operation efficiency.

Description

Technical Field

[0001] This application relates to the technical field of material handling control, specifically to a visual feedback material handling control method and device applicable to hardware parts of various specifications. Background Technology

[0002] In the field of hardware manufacturing and automated assembly, automated picking and sorting of multi-specification hardware parts is a critical link in the production line. Traditional picking relies on preset programs or single vision guidance, which is difficult to adapt to the precise grasping requirements of hardware parts with different specifications, shapes, and stacking states. Especially under the influence of factors such as diverse hardware parts sizes, random stacking postures, and ambient light interference, the stability and accuracy of picking are poor. In addition, due to the reflective properties of hardware parts and complex edge features, visual recognition based on two-dimensional images often struggles to achieve high-precision pose estimation, resulting in a high picking failure rate and affecting production efficiency. While picking based on multi-light source vision and real-time feedback can ensure the accuracy and stability of picking, there are still shortcomings in the rapid identification of multi-specification hardware parts, dynamic path correction, and adaptive control of fixtures. For example, the mixed stacking of hardware parts of different specifications may lead to point cloud segmentation errors, and dynamic deviations in the movement of the robotic arm may further increase the risk of grasping failure.

[0003] Therefore, current technologies suffer from insufficient accuracy in visual inspection and recognition of multi-specification hardware parts and weak real-time correction capability of dynamic material picking paths, resulting in poor sorting accuracy and operational efficiency for multi-specification hardware parts. Summary of the Invention

[0004] This application provides a visual feedback material handling control method and device applicable to multi-specification hardware parts, which solves the technical problems in the prior art of insufficient visual inspection and recognition accuracy of multi-specification hardware parts and weak real-time correction capability of dynamic material handling path, resulting in poor sorting accuracy and operation efficiency of multi-specification hardware parts. It achieves the technical effect of improving the visual inspection and recognition accuracy, sorting accuracy and operation efficiency of multi-specification hardware parts.

[0005] This application provides a visual feedback material handling control method applicable to multi-specification hardware parts. The method includes: acquiring real-time images of the stacked area of ​​hardware parts through a multi-light source vision unit to generate three-dimensional point cloud data; performing multi-specification hardware part segmentation and recognition based on the three-dimensional point cloud data to determine the specification type information of the parts to be handled; performing material handling analysis based on the specification type information according to the spatial pose of the multi-specification hardware parts to generate a material handling parameter set, the material handling parameter set including material handling path parameters and material handling fixture parameters; executing the material handling path parameters for real-time visual tracking feedback to draw a pose change trend map of the multi-specification hardware parts; calculating the deviation of the material handling path parameters based on the pose change trend map; correcting the material handling trajectory according to the path deviation; and optimizing the control based on the correction result and the material handling fixture parameters to formulate a material handling control scheme.

[0006] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: the multi-light source vision unit uses a combination of coaxial light source and ring light source for illumination, activates a binocular stereo camera to synchronously acquire data from multiple perspectives of the stacked hardware parts area, and obtains a multi-perspective original image set; stereo matching is performed based on the multi-perspective original image set to generate an initial three-dimensional point cloud; metal reflection compensation of the hardware parts is performed according to the initial three-dimensional point cloud to determine photometric stereo fusion data; the initial three-dimensional point cloud is enhanced according to the photometric stereo fusion data to determine the three-dimensional point cloud data.

[0007] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: performing geometric analysis on the multi-specification hardware parts according to the three-dimensional point cloud data to generate geometric feature vectors; performing contour recognition on the multi-specification hardware parts based on the geometric feature vectors to determine contour parameters; performing clustering and segmentation on the multi-specification hardware parts according to the contour parameters to obtain multiple independent hardware part structural parameters; and performing specification recognition based on the multiple independent hardware part structural parameters combined with the three-dimensional point cloud data to determine the specification type information of the part to be handled.

[0008] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: arranging the three-dimensional point cloud data according to the structural parameters of the multiple independent hardware parts to construct a point cloud normal vector histogram; calculating the point cloud curvature based on the point cloud normal vector histogram, extracting extreme values ​​according to the calculation results to obtain point cloud curvature extreme points; using the point cloud curvature extreme points as seed points of the point cloud normal vector histogram for region growth to obtain growth region volume data; when the growth region volume reaches a preset minimum volume threshold, terminating the region growth of the seed points, extracting the growth region volume to classify the structural parameters of the multiple independent hardware parts according to specifications, and determining the specification type information of the part to be handled.

[0009] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: analyzing the material handling fixture based on the specification type information to determine the fixture angle parameters; analyzing the material handling motion of the material handling device according to the fixture angle parameters to determine the motion speed parameters; analyzing the material handling pose according to the spatial pose of the multi-specification hardware parts in conjunction with the specification type information to determine the pose tolerance parameters; simulating the material handling and clamping of the multi-specification hardware parts based on the fixture angle parameters to determine the material handling fixture parameters; and analyzing the material handling trajectory based on the motion speed parameters in conjunction with the pose tolerance parameters to determine the material handling path parameters.

[0010] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: performing pose rotation analysis based on the pose tolerance parameters to construct a pose matrix; performing material handling accessibility analysis on the pose matrix based on the motion speed parameters; when the pose matrix is ​​material handling accessible, extracting the material handling trajectory to determine the material handling path parameters; when the pose matrix is ​​material handling unreachable, triggering a repositioning command, retrieving the pose matrix through the repositioning command, retrieving the motion speed parameters to verify the retrieval results, updating the pose matrix until the pose matrix is ​​material handling accessible, and extracting the material handling path parameters.

[0011] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: executing the material handling path parameters, activating the high-speed vision unit to track the material handling device, and obtaining a dynamic image sequence of material handling for multi-specification hardware parts; performing adjacent frame matching on the pose matrix according to the dynamic image sequence of material handling, and extracting multiple pose adjacent frame data; performing displacement calculation based on the multiple pose adjacent frame data to obtain an adjacent frame pose change matrix; performing trend identification based on the adjacent frame pose change matrix according to the change time sequence, connecting the identification results adjacently, and drawing the pose change trend map.

[0012] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: analyzing the pose change trend map to obtain six-degree-of-freedom pose timing data; determining a bi-branch trajectory based on the six-degree-of-freedom pose timing data and the material handling path parameters to generate a trajectory determination result, the trajectory determination result including a collision determination result or a deviation determination result; when the trajectory determination result is the collision determination result, marking the material handling path parameters with an intersection marker according to the six-degree-of-freedom pose timing data to determine a path collision point, and adding the path collision point to the path deviation amount; when the trajectory determination result is the deviation determination result, marking the material handling path parameters with a moving-away progressive marker according to the six-degree-of-freedom pose timing data to determine the deviation progressive data, backtracking the material handling path parameters according to the deviation progressive data to determine the deviation starting point; adding the deviation progressive data and the deviation starting point to the path deviation amount.

[0013] In a possible implementation, the visual feedback material handling control method applicable to multi-specification hardware parts further performs the following processing: traversing and matching the material handling path parameters based on the six-degree-of-freedom pose data to generate a pose path matching result; performing distance continuous analysis based on the pose path matching result to obtain a distance gradient parameter; generating the deviation determination result when the distance gradient parameter gradually increases; and generating the collision determination result when the distance gradient parameter gradually decreases.

[0014] This application also provides a visual feedback material handling control device applicable to multi-specification hardware parts. The device includes: a hardware part segmentation and recognition module, used to acquire real-time images of the hardware part stacking area through a multi-light source vision unit, generate three-dimensional point cloud data, and perform multi-specification hardware part segmentation and recognition based on the three-dimensional point cloud data to determine the specification type information of the part to be handled; a material handling analysis module, used to perform material handling analysis based on the spatial pose of the multi-specification hardware parts and the specification type information to generate a material handling parameter set, the material handling parameter set including material handling path parameters and material handling fixture parameters; a visual tracking feedback module, used to perform real-time visual tracking feedback based on the material handling path parameters and draw a pose change trend diagram of the multi-specification hardware parts; and a material handling path deviation calculation module, used to calculate the deviation of the material handling path parameters based on the pose change trend diagram, correct the material handling trajectory according to the path deviation, and optimize the control based on the correction result and the material handling fixture parameters to formulate a material handling control scheme.

[0015] This application proposes a visual feedback material handling control method and device applicable to multi-specification hardware parts. The method involves acquiring real-time images of the stacked hardware parts area using a multi-light source vision unit, performing multi-specification hardware part segmentation and recognition based on 3D point cloud data, analyzing material handling based on specification type information to generate a material handling parameter set, executing material handling path parameters for real-time visual tracking feedback, calculating deviations in the material handling path parameters based on pose change trend maps, correcting the material handling trajectory based on the path deviation, and optimizing control by combining material handling fixture parameters to formulate a material handling control scheme. This solves the technical problems in existing technologies, such as insufficient visual inspection and recognition accuracy for multi-specification hardware parts and weak real-time correction capability of dynamic material handling paths, leading to poor sorting accuracy and operational efficiency for multi-specification hardware parts. It achieves the technical effect of improving the visual inspection and recognition accuracy, sorting accuracy, and operational efficiency of multi-specification hardware parts. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the apparatus according to the embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a schematic flowchart of a visual feedback material handling control method applicable to multi-specification hardware parts, provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of a visual feedback material handling control device for multi-specification hardware parts provided in an embodiment of this application.

[0019] Figure labeling: Hardware part segmentation and recognition module 10, material picking analysis module 20, visual tracking feedback module 30, material picking path deviation calculation module 40. Detailed Implementation

[0020] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. The terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or apparatuses. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0023] This application provides a visual feedback material handling control method applicable to multi-specification hardware parts, such as... Figure 1 As shown, the method includes:

[0024] Step S100: Real-time images of the stacked area of ​​hardware parts are acquired by a multi-light source vision unit to generate three-dimensional point cloud data. Based on the three-dimensional point cloud data, multi-specification hardware parts are segmented and identified to determine the specification type information of the parts to be taken.

[0025] Preferably, the hardware components in the stacked area may be mixed and stacked, such as bolts, nuts, and washers of different specifications. Acquiring data from a single light source can easily lead to poor image quality due to problems such as metal reflection and shadow occlusion. By using a multi-light source vision unit, such as structured light, LED array, or multi-angle industrial camera, controllable light is projected from different directions to ensure that the surface features such as the edges, grooves, and textures of the hardware components are fully illuminated, reducing reflection interference. This allows for the acquisition of high-contrast, low-noise real-time images of the stacked hardware components. Based on the multi-view images, the depth information of the hardware component surface is calculated through stereo vision or structured light 3D reconstruction to generate 3D point cloud data. The 3D point cloud data consists of a large number of 3D coordinate points (X, Y, Z), which can accurately characterize the geometry, spatial position, and surface contour of the hardware components.

[0026] Preferably, multi-specification hardware parts segmentation and recognition is performed based on 3D point cloud data. This involves contour segmentation, target recognition, and feature extraction of the point cloud data. Specifically, point cloud data preprocessing is performed, including filtering and denoising, statistical outlier removal, and downsampling to reduce computational load. Then, clustering algorithms such as Euclidean clustering and DBSCAN are used to segment the point cloud contours, dividing the point cloud into different object instances. Based on the point cloud contour segmentation results, the geometric features of each hardware part, such as size and volume, are extracted. Combined with template matching, the specification type information of the part to be picked is determined, including category and size, which can improve the sorting efficiency and reliability of automated production lines.

[0027] Furthermore, step S100 also includes step S110, whereby the multi-source vision unit uses a combination of coaxial light source and ring light source for illumination, activates the binocular stereo camera to perform multi-view synchronous acquisition of the stacked area of ​​the hardware parts, and obtains a multi-view original image set; step S120, performs stereo matching based on the multi-view original image set to generate an initial three-dimensional point cloud; step S130, performs metal reflection compensation on the hardware parts according to the initial three-dimensional point cloud to determine the photometric stereo fusion data; step S140, enhances the initial three-dimensional point cloud according to the photometric stereo fusion data to determine the three-dimensional point cloud data.

[0028] Preferably, the multi-light source vision unit uses a combination of coaxial and ring light sources for illumination. The coaxial light source provides axial parallel light, projecting uniform light along the camera's optical axis. This is mainly used to eliminate overexposure or highlight interference caused by specular reflection on the surface of the hardware, ensuring that the texture of planar areas such as screw heads and washer surfaces is clearly visible. The ring light source provides multi-angle diffused light, arranged around the lens, providing soft light with oblique incidence, highlighting three-dimensional features such as threads and edges of the hardware, such as edges and grooves, and reducing shadow occlusion. The binocular stereo camera is activated to capture images of the stacked hardware area from different perspectives, obtaining a multi-view original image set of the stacked hardware. Then, based on stereo matching of binocular vision, such as a deep learning stereo matching network, the correspondence of the same hardware feature points is found in the multi-view images. Through the principle of triangulation, the pixel displacement of the matching point pair is converted into a depth value. Combined with camera calibration parameters, the depth information is fused with the two-dimensional image coordinates to output an initial three-dimensional point cloud, including XYZ coordinates and initial RGB color information.

[0029] Preferably, surface reflections on metal may cause matching errors, and the initial point cloud may contain voids or noise. Photometric stereo vision is used to enhance the initial 3D point cloud to compensate for metal reflections on the hardware. Specifically, by calculating the reflection intensity changes of coaxial and ring light at different illumination angles, the surface normal direction of the hardware is calculated, and high-precision surface material information, such as diffuse or specular reflection components, is reconstructed to obtain a normal map. Abnormal brightness spots caused by specular reflection are then removed, and missing surface geometric information is interpolated using multi-source data to determine the texture map. Next, the normal map and the texture map after reflection suppression are aligned and fused with the initial point cloud to obtain photometric stereo fusion data. Finally, the initial 3D point cloud is enhanced based on the photometric stereo fusion data, including point cloud repair, noise filtering, and detail enhancement. Specifically, voids in the point cloud caused by reflections, such as the highlight area of ​​the screw head, are filled using photometric stereo data; outlier noise is removed using statistical filtering; and multi-source texture information is fused to enhance key features such as the thread profile and stamping marks of the hardware, ultimately outputting high-precision, complete 3D point cloud data.

[0030] Furthermore, step S100 also includes step S150, performing geometric analysis on the multi-specification hardware parts according to the three-dimensional point cloud data to generate geometric feature vectors; step S160, performing contour recognition on the multi-specification hardware parts based on the geometric feature vectors to determine contour parameters; step S170, performing clustering and segmentation on the multi-specification hardware parts according to the contour parameters to obtain multiple independent hardware part structural parameters; and step S180, performing specification recognition based on the multiple independent hardware part structural parameters combined with the three-dimensional point cloud data to determine the specification type information of the part to be taken.

[0031] Preferably, geometric analysis is performed on multi-specification hardware parts based on 3D point cloud data to extract key features of each hardware part, including size, volume, surface area, curvature distribution, principal axis direction, edge sharpness, number of holes, etc., and these are described as a set of values ​​to determine the geometric feature vector. Then, based on the geometric feature vector, edge detection or projection contour analysis is used to identify the outer contour and internal structural features of the hardware parts, and parameters such as diameter, roundness, concavity / convexity, and key point distribution are calculated to form contour parameters. Since hardware parts may be stacked or partially occluded, the contour parameters are used to analyze the multi-specification hardware parts. The process involves point cloud clustering and segmentation, such as using Euclidean clustering or region growing algorithms to separate the point clouds of different hardware components, and calculating the structural parameters of each independent hardware component, including pose, size, and shape features. Based on the structural parameters of multiple independent hardware components combined with 3D point cloud data, specification identification is performed. This involves matching the segmented hardware component structural parameters with a predefined specification database, which contains the standard dimensions of various hardware components. Then, the specification type of the hardware component is determined through nearest neighbor search. Finally, the precise category and pose information of the component to be retrieved are output, thus determining the specification type information of the component to be retrieved.

[0032] Furthermore, step S180 also includes step S181, arranging the three-dimensional point cloud data according to the structural parameters of the multiple independent hardware parts to construct a point cloud normal vector histogram; step S182, calculating the point cloud curvature based on the point cloud normal vector histogram, extracting extreme values ​​according to the calculation results, and obtaining the point cloud curvature extreme points; step S183, using the point cloud curvature extreme points as seed points of the point cloud normal vector histogram for region growth, and obtaining the growth region volume data; step S184, when the growth region volume reaches a preset minimum volume threshold, terminating the region growth of the seed points, extracting the growth region volume to classify the specifications of the multiple independent hardware part structural parameters, and determining the specification type information of the part to be taken.

[0033] Preferably, the 3D point cloud data is arranged according to the structural parameters of multiple independent hardware parts. That is, the normal vector of the segmented independent hardware part point cloud data is statistically calculated. The normal vector direction of each point is estimated by principal component analysis, and the azimuth angle of the normal vector in the spherical coordinate system is discretized and statistically analyzed to construct a 3D point cloud normal vector histogram, which effectively represents the geometric orientation characteristics of the hardware part surface. For example, planar areas show a concentrated distribution of normal vectors, curved or edge areas show a dispersed distribution of normal vectors, and periodic structures such as threads show a regular distribution pattern. Then, the point cloud curvature is calculated based on the point cloud normal vector histogram. That is, differential geometric calculation is performed based on the normal vector histogram to derive the point cloud curvature characteristics. By quantitatively analyzing parameters such as Gaussian curvature and average curvature, the extreme points of point cloud curvature are extracted, including curvature maxima and curvature minima. These usually correspond to key feature parts of the hardware part, such as the edge of the bolt head, the corner of the nut, the boundary of the gasket hole, and the peaks and troughs of the thread.

[0034] Preferably, the extreme points of point cloud curvature are used as seed points for the point cloud normal vector histogram for region growth. Specifically, spatial proximity (Euclidean distance) and geometric similarity (angle between normal vectors) are considered as growth conditions. Then, feature extraction is performed to obtain the volume data of the growth region, including recording parameters such as volume, surface area, and aspect ratio of each growth region. The growth is stopped when the volume of the growth region reaches a preset minimum volume threshold, for example, the growth of the seed point is terminated when the growth region reaches 50% of the minimum hardware part volume. Then, the volume of the growth region is extracted to classify the specifications of multiple independent hardware parts, including matching the volume features of the growth region with a preset specification library. For example, a large volume area at the head and a slender area at the rod may be a bolt, a special volume distribution due to a central hole may be a nut, and a thin sheet-like volume feature may be a gasket. Finally, the specification type information of the part to be picked is determined, thereby ensuring accurate and efficient identification of hardware parts in complex stacking states.

[0035] Step S200: Based on the spatial orientation of the multi-specification hardware parts, perform material picking analysis according to the specification type information to generate a material picking parameter set, which includes material picking path parameters and material picking fixture parameters.

[0036] Step S200 further includes step S210, performing a material handling fixture analysis on the material handling equipment based on the specification type information to determine the fixture angle parameters; step S220, performing a material handling motion analysis on the material handling equipment according to the fixture angle parameters to determine the motion speed parameters; step S230, performing a material handling pose analysis based on the spatial pose of the multi-specification hardware parts and the specification type information to determine the pose tolerance parameters; step S240, performing a material handling clamping simulation of the multi-specification hardware parts based on the fixture angle parameters to determine the material handling fixture parameters; and step S250, performing a material handling trajectory analysis based on the motion speed parameters and the pose tolerance parameters to determine the material handling path parameters.

[0037] Preferably, a multi-dimensional parameter analysis is performed to generate a set of material handling parameters, ensuring the accuracy and reliability of the material handling operation. Specifically, the material handling equipment is analyzed based on specification type information, including calculating the opening angle of the clamping jaws according to the geometric characteristics of the target part, and automatically selecting the clamping type for different specification types. For example, a parallel clamp is used for regular cubic parts, a three-jaw clamp is used for cylindrical nuts, and a vacuum suction cup is used for flat thin hardware parts, thereby determining the clamping angle parameters. At the same time, an operating margin of 2° to 5° is added to the calculated angle to avoid damaging the parts. Then, the material handling motion of the material handling equipment is analyzed according to the clamping angle parameters. That is, the optimal motion parameters are determined through mechanical simulation, including high-speed operation (1-2m / s) to 50mm from the workpiece, and then deceleration to 0.1-0.3m / s for final precise positioning. The acceleration curve is automatically adjusted according to the weight of the hardware part for load compensation.

[0038] Preferably, the material handling posture analysis is performed based on the spatial pose of multi-specification hardware parts combined with specification type information. That is, by comprehensively considering the actual spatial pose of the hardware parts, specification type characteristics, and surrounding environmental constraints such as the distance between adjacent parts and stacking stability, a six-degree-of-freedom error model is established to quantify the tolerance range and determine the allowable position and angle deviation range of the robotic arm end effector gripper during grasping. This is used as the posture tolerance parameter. Possible posture tolerance parameter data are shown in Table 1.

[0039] Table 1. Data Table of Posture Tolerance Parameters

[0040] ;

[0041] Preferably, the material handling and clamping simulation of multi-specification hardware parts is performed based on clamping angle parameters. This involves converting specification identification results into executable clamping parameters through virtual simulation. This includes automatically matching clamping types according to hardware specifications, verifying the contact stress distribution on the clamping surface using a material mechanics model to prevent deformation of precision parts, establishing a clamping-workpiece coordinate system transformation matrix, calculating the optimal clamping angle, and calculating the minimum clamping force based on the hardware weight and surface friction coefficient, thereby determining the material handling clamping parameters. Finally, the material handling trajectory is analyzed based on motion speed parameters combined with posture tolerance parameters. Specifically, the posture tolerance parameters are converted into velocity constraints, and a dynamic adjustment zone is set in the critical path segment. A positional deviation of ±3mm is allowed during the approach phase and reduced to ±0.5mm during the precision positioning phase. A cost function is then established to optimize the trajectory, balancing time efficiency, motion smoothness, and positioning accuracy, ultimately obtaining the material handling path parameters. This improves the visual inspection and recognition accuracy, sorting accuracy, and operational efficiency of multi-specification hardware parts, achieving intelligent and flexible handling operations.

[0042] Furthermore, step S250 also includes step S251, performing pose rotation analysis based on the pose tolerance parameters to construct a pose matrix; step S252, performing material accessibility analysis on the pose matrix based on the motion velocity parameters, and extracting the material access trajectory to determine the material access path parameters when the pose matrix is ​​material accessible; step S253, triggering a relocation command when the pose matrix is ​​material access inaccessible, retrieving the pose matrix through the relocation command, verifying the retrieval results by retrieving the motion velocity parameters, updating the pose matrix until the pose matrix is ​​material accessible, and extracting the material access path parameters.

[0043] Preferably, based on the pose tolerance parameters of the hardware part, including position coordinates, rotation angles, and specification tolerance parameters, a pose matrix is ​​constructed using rigid body transformation theory. Homogeneous coordinate representation is used to transform the spatial state of the target part into a transformation matrix. Position tolerance is converted into the allowable fluctuation range of translation in the matrix, angle tolerance is reflected in the acceptable range of eigenvalues ​​of the rotation matrix, and specification features are labeled through matrix attribute annotations. This establishes a pose matrix containing the tolerance range. Then, based on motion velocity parameters, the pose matrix is ​​analyzed for material accessibility. Specifically, inverse kinematics calculations are used to determine whether the target pose is within the angular limits of each joint of the robotic arm. The temporal-spatial feasibility of the trajectory segment is verified based on the motion velocity parameters, while also considering the limitation of the hardware part's mass distribution on the maximum acceleration of the end effector. If the pose matrix meets the verification criteria, material accessibility is determined, and the material access trajectory is extracted, generating material access path parameters including a path point sequence, velocity curve, and acceleration curve.

[0044] Preferably, when the initial pose matrix does not meet the reachability conditions, a relocation command is triggered for intelligent relocation. Specifically, within the pose tolerance range, a candidate pose matrix set is generated by searching with a position step size of 0.1 mm and an angle step size of 0.5°. The motion velocity parameters are called to pre-verify the candidate poses, and obviously unreachable solutions are eliminated. Then, the nearest reachable pose is found from the original pose along the gradient direction. The pose matrix is ​​updated through iterative optimization until the pose matrix is ​​reachable for material picking, and the corrected feasible pose matrix is ​​obtained. The updated material picking path parameters are extracted to realize closed-loop control from visual perception to physical execution, thereby improving the visual inspection and recognition accuracy, sorting accuracy and operation efficiency of multi-specification hardware parts.

[0045] Step S300: Execute the material picking path parameters to perform real-time visual tracking feedback and draw a trend diagram of the position and pose changes of hardware parts of various specifications.

[0046] Step S300 further includes step S310, executing the material picking path parameters, activating the high-speed vision unit to track the material picking device, and obtaining a dynamic image sequence of material picking for multiple specifications of hardware parts; step S320, performing adjacent frame matching on the pose matrix according to the material picking dynamic image sequence, and extracting multiple pose adjacent frame data; step S330, performing displacement calculation based on the multiple pose adjacent frame data, and obtaining an adjacent frame pose change matrix; step S340, performing trend identification based on the adjacent frame pose change matrix according to the change time sequence, connecting the identification results adjacently, and drawing the pose change trend map.

[0047] Preferably, real-time visual tracking feedback is performed on the material handling path parameters. High-frame-rate image sequence analysis enables accurate tracking and trend prediction of the hardware's motion trajectory. Specifically, when the robotic arm handles the hardware handling according to the path parameters, a high-speed vision unit, such as a high-speed industrial camera, is activated to track the material handling equipment. Simultaneously, dynamic images of hardware handling of various specifications are acquired, and temporal alignment is performed to determine the dynamic image sequence. Then, adjacent frames of the pose matrix are matched according to the dynamic image sequence. Specifically, a dual matching strategy at the feature level and region level is adopted to detect stable feature points on the hardware and fixtures. A random sampling consensus algorithm iteratively randomizes data from data containing a large amount of noise or outliers, eliminating mismatched points and extracting multiple adjacent pose frame data to establish feature correspondences between adjacent frames. Then, displacement calculation is performed based on multiple adjacent pose frame data, i.e., the relative pose change between adjacent frames is calculated based on geometric principles, ultimately outputting the adjacent frame pose change matrix. Based on the pose change matrix of adjacent frames, the trend is marked according to the change time sequence. That is, the change curves are drawn in the six-dimensional pose space, the main direction of motion is extracted, and abnormal fluctuations are marked. Finally, the pose change trend map is drawn with the horizontal axis as the standardized time and the vertical axis as the position deviation magnitude and attitude change angle. When the pose change trend map continues to deviate or the angular velocity change rate exceeds the safety value, the compensation command is automatically generated. The final material picking pose accuracy is ensured by fine adjustment of the robotic arm joints.

[0048] Step S400: Based on the pose change trend map, the deviation of the material picking path parameters is calculated, the material picking trajectory is corrected according to the path deviation, and the control is optimized according to the correction result and the material picking fixture parameters to formulate a material picking control scheme.

[0049] Step S400 further includes step S410, analyzing the pose change trend map to obtain six-degree-of-freedom pose timing data; step S420, performing bi-branch trajectory determination based on the six-degree-of-freedom pose timing data and the material handling path parameters to generate a trajectory determination result, the trajectory determination result including a collision determination result or a deviation determination result; step S430, when the trajectory determination result is the collision determination result, marking the material handling path parameters with an intersection marker according to the six-degree-of-freedom pose timing data to determine the path collision point, and adding the path collision point to the path deviation amount; step S440, when the trajectory determination result is the deviation determination result, marking the material handling path parameters with a moving-away progressive marker according to the six-degree-of-freedom pose timing data to determine the deviation progressive data, and backtracking the material handling path parameters according to the deviation progressive data to determine the deviation starting point; step S450, adding the deviation progressive data and the deviation starting point to the path deviation amount.

[0050] Preferably, the pose change trend map is analyzed to extract the complete spatial state at each time point, i.e., to obtain accurate six-degree-of-freedom pose time-series data, including three-dimensional position, three-dimensional attitude, and corresponding timestamps. Then, based on the six-degree-of-freedom pose time-series data, a dual-branch trajectory determination is performed according to the material handling path parameters. That is, through parallel collision determination and deviation determination analysis, the material handling process is ensured to be safe and reliable. Specifically, the geometric relationship between the actuator (gripper / robotic arm) and the surrounding environment is compared in real time, and the potential collision contact points in the next 3-5 control cycles are predicted based on the kinematic model. When the minimum distance detected is less than the safety threshold, a collision determination result including the collision position and collision object ID is generated. The actual pose data and theoretical path parameters are spatiotemporally synchronized to calculate the position deviation and attitude deviation, identify the offset pattern, and when the deviation of any degree of freedom exceeds the tolerance range, a deviation determination result including the deviation direction and deviation rate is generated.

[0051] Preferably, when the trajectory determination result is a collision determination result, that is, when a collision risk is determined, the first contact position between the robotic arm / gripper and the obstacle is determined by spatial geometric calculation based on the six-degree-of-freedom pose data. The precise coordinates of the point and the corresponding time node are marked in the three-dimensional path model. At the same time, collision feature analysis is performed, and the normal vector direction of the collision surface is recorded. For example, when a collision is detected between the side of the gripper and the column of the rack, the avoidance direction that needs to be adjusted radially along the column is marked. Then, the spatial coordinates of the collision point, the contact angle, and the suggested avoidance vector are packaged into structured deviation data and added to the path deviation amount to guide the path planning to generate a safe buffer distance in this area, which is usually set to 1.2 to 1.5 times the detection distance.

[0052] Preferably, when the trajectory determination result is a deviation determination result, i.e., a deviation risk is determined, the material handling path parameters are marked with a progressive deviation indicator based on the six-degree-of-freedom pose time series data. Specifically, the deviation rate and acceleration of the pose parameters relative to the theoretical path are calculated through time series analysis to establish a deviation dynamic model. Then, root cause backtracking is performed, i.e., reverse analysis along the time axis, and correlation analysis is performed on the operating parameters at that moment, such as sudden changes in joint motor current and external vibration interference, to determine the precise moment when the deviation first exceeds the allowable threshold, which is defined as the deviation start point. The deviation direction, deviation rate, and maximum deviation amount are then integrated into the deviation progressive data. Finally, the deviation progressive data and the deviation start point are added to the path deviation amount to ensure the reliability of hardware material handling.

[0053] Preferably, the material handling trajectory is corrected based on the path deviation. Specifically, for random fluctuations, the pose of the next cycle is predicted, and feedforward compensation instructions are generated. When a continuous unidirectional deviation is detected, the coordinates of key points on the path are reconstructed. The natural frequency of the robotic arm is identified through frequency domain analysis, and the trajectory acceleration curve is optimized to avoid dangerous speed ranges. The corrected path parameters are obtained and virtually verified through a digital twin platform to ensure that the new trajectory meets all constraints. Then, control optimization is performed based on the correction results and the material handling fixture parameters. This includes adjusting the clamping parameters according to the spatial characteristics of the corrected path. For example, for a corrected trajectory with a 15° inclination, the contact area of ​​the gripper is automatically increased by 30%. Kinematic parameters are adjusted based on the latest deviation statistics, such as reducing the maximum acceleration in the high-speed section from 2 m / s² to 1.5 m / s² to suppress vibration. Finally, a three-dimensional working space boundary with safety margins is generated, and the material handling control scheme is determined, thereby improving the visual inspection and recognition accuracy, sorting accuracy, work efficiency, and material handling quality of multi-specification hardware parts.

[0054] Furthermore, step S420 also includes step S421, performing traversal matching on the material picking path parameters based on the six-degree-of-freedom pose data to generate a pose path matching result; step S422, performing distance continuous analysis based on the pose path matching result to obtain a distance gradient parameter; step S423, generating the deviation determination result when the distance gradient parameter gradually increases; and step S424, generating the collision determination result when the distance gradient parameter gradually decreases.

[0055] Preferably, the material handling path parameters are traversed and matched based on six-degree-of-freedom pose data. Specifically, spatial topology matching is used to compare the real-time acquired six-degree-of-freedom pose data with the preset material handling path parameters point by point. The actual pose is mapped to the corresponding nodes of the theoretical path through timestamp synchronization. Simultaneously, positional and attitude deviations are calculated to generate a pose path matching result. Then, based on the pose path matching result, continuous distance analysis is performed, including continuously measuring the shortest spatial distance from the actual pose point to the theoretical path. The derivative characteristics of distance changes are calculated using a sliding window algorithm. A continuously increasing positive gradient distance indicates a gradual deviation from the path; a continuously decreasing negative gradient distance indicates approaching an obstacle. When the distance gradually increases, the deviation starting point position is recorded, the deviation angular velocity is calculated, and a deviation judgment result is generated. When the distance gradually decreases, the nearest obstacle is marked, the remaining buffer distance is calculated, and the collision occurrence time is estimated, ultimately generating a collision judgment result.

[0056] In the above text, refer to Figure 1 A visual feedback material handling control method for multi-specification hardware parts, according to an embodiment of the present invention, is described in detail. Next, reference will be made to... Figure 2 This invention describes a visual feedback material handling control device suitable for multi-specification hardware parts according to an embodiment of the present invention.

[0057] The visual feedback material handling control device for multi-specification hardware parts according to embodiments of the present invention solves the technical problems in the prior art, such as insufficient visual inspection and recognition accuracy and weak real-time correction capability of dynamic material handling paths for multi-specification hardware parts, resulting in poor sorting accuracy and work efficiency of multi-specification hardware parts. It achieves the technical effect of improving the visual inspection and recognition accuracy, sorting accuracy, and work efficiency of multi-specification hardware parts. Figure 2 As shown, the visual feedback material handling control device applicable to multi-specification hardware parts includes: a hardware part segmentation and recognition module 10, a material handling analysis module 20, a visual tracking feedback module 30, and a material handling path deviation calculation module 40.

[0058] The hardware component segmentation and recognition module 10 is used to acquire real-time images of the hardware component stacking area through a multi-light source vision unit, generate three-dimensional point cloud data, and perform multi-specification hardware component segmentation and recognition based on the three-dimensional point cloud data to determine the specification type information of the component to be picked up; the material picking analysis module 20 is used to perform material picking analysis based on the spatial pose of the multi-specification hardware components and the specification type information to generate a material picking parameter set, which includes material picking path parameters and material picking fixture parameters; the visual tracking feedback module 30 is used to perform real-time visual tracking feedback based on the material picking path parameters and draw a pose change trend diagram of the multi-specification hardware components; the material picking path deviation calculation module 40 is used to calculate the deviation of the material picking path parameters based on the pose change trend diagram, correct the material picking trajectory according to the path deviation, and optimize the control based on the correction result and the material picking fixture parameters to formulate a material picking control scheme.

[0059] The specific configuration of the hardware component segmentation and recognition module 10 will be described in detail below. The hardware component segmentation and recognition module 10 further includes: the multi-light source vision unit uses a combination of coaxial and ring light sources for illumination, activates a binocular stereo camera to simultaneously acquire data from multiple perspectives of the stacked hardware component area, and obtains a multi-perspective original image set; performs stereo matching based on the multi-perspective original image set to generate an initial three-dimensional point cloud; performs metal reflection compensation on the hardware component according to the initial three-dimensional point cloud to determine photometric stereo fusion data; and enhances the initial three-dimensional point cloud according to the photometric stereo fusion data to determine the three-dimensional point cloud data.

[0060] The specific configuration of the hardware component segmentation and recognition module 10 will be described in detail below. The hardware component segmentation and recognition module 10 further includes: performing geometric analysis on multi-specification hardware components according to the three-dimensional point cloud data to generate geometric feature vectors; performing contour recognition on the multi-specification hardware components based on the geometric feature vectors to determine contour parameters; performing clustering and segmentation on the multi-specification hardware components according to the contour parameters to obtain multiple independent hardware component structural parameters; and performing specification recognition based on the multiple independent hardware component structural parameters combined with the three-dimensional point cloud data to determine the specification type information of the component to be retrieved.

[0061] The specific configuration of the hardware component segmentation and recognition module 10 will be described in detail below. The hardware component segmentation and recognition module 10 further includes: arranging the three-dimensional point cloud data according to the multiple independent hardware component structural parameters to construct a point cloud normal vector histogram; calculating the point cloud curvature based on the point cloud normal vector histogram, extracting extreme values ​​based on the calculation results, and obtaining point cloud curvature extreme points; using the point cloud curvature extreme points as seed points of the point cloud normal vector histogram for region growth to obtain growth region volume data; terminating the region growth of the seed points when the growth region volume reaches a preset minimum volume threshold, extracting the growth region volume to classify the multiple independent hardware component structural parameters according to specifications, and determining the specification type information of the component to be retrieved.

[0062] The specific configuration of the material handling analysis module 20 will be described in detail below. The material handling analysis module 20 further includes: performing material handling fixture analysis on the material handling equipment based on the specification type information to determine fixture angle parameters; performing material handling motion analysis on the material handling equipment according to the fixture angle parameters to determine motion speed parameters; performing material handling pose analysis based on the spatial pose of the multi-specification hardware parts combined with the specification type information to determine pose tolerance parameters; performing material handling clamping simulation on the multi-specification hardware parts based on the fixture angle parameters to determine the material handling fixture parameters; and performing material handling trajectory analysis based on the motion speed parameters combined with the pose tolerance parameters to determine material handling path parameters.

[0063] The specific configuration of the material handling analysis module 20 will be described in detail below. The material handling analysis module 20 further includes: performing pose rotation analysis based on the pose tolerance parameters to construct a pose matrix; performing material handling accessibility analysis on the pose matrix based on the motion speed parameters; when the pose matrix is ​​material handling accessible, extracting the material handling trajectory to determine the material handling path parameters; when the pose matrix is ​​material handling inaccessible, triggering a relocation command, retrieving the pose matrix through the relocation command, retrieving the motion speed parameters to verify the retrieval results, updating the pose matrix until the pose matrix is ​​material handling accessible, and extracting the material handling path parameters.

[0064] The specific configuration of the visual tracking feedback module 30 will be described in detail below. The visual tracking feedback module 30 further includes: executing the material handling path parameters, activating the high-speed vision unit to track the material handling equipment, and obtaining a dynamic image sequence of material handling for multiple specifications of hardware parts; performing adjacent frame matching on the pose matrix according to the dynamic image sequence of material handling, and extracting multiple adjacent pose frame data; performing displacement calculation based on the multiple adjacent pose frame data to obtain an adjacent frame pose change matrix; and performing trend identification based on the adjacent frame pose change matrix according to the change sequence, connecting the identification results adjacently, and drawing the pose change trend map.

[0065] The specific configuration of the material handling path deviation calculation module 40 will be described in detail below. The material handling path deviation calculation module 40 further includes: analyzing the pose change trend map to obtain six-degree-of-freedom pose timing data; performing bi-branch trajectory determination based on the six-degree-of-freedom pose timing data and the material handling path parameters to generate a trajectory determination result, the trajectory determination result including a collision determination result or a deviation determination result; when the trajectory determination result is the collision determination result, marking the material handling path parameters with an intersection marker according to the six-degree-of-freedom pose timing data to determine a path collision point, and adding the path collision point to the path deviation amount; when the trajectory determination result is the deviation determination result, marking the material handling path parameters with a moving-away progressive marker according to the six-degree-of-freedom pose timing data to determine the deviation progressive data, backtracking the material handling path parameters according to the deviation progressive data to determine the deviation starting point; and adding the deviation progressive data and the deviation starting point to the path deviation amount.

[0066] The specific configuration of the material picking path deviation calculation module 40 will be described in detail below. The material picking path deviation calculation module 40 further includes: performing traversal matching on the material picking path parameters based on the six-degree-of-freedom pose data to generate a pose path matching result; performing distance continuous analysis based on the pose path matching result to obtain a distance gradient parameter; generating the deviation determination result when the distance gradient parameter gradually increases; and generating the collision determination result when the distance gradient parameter gradually decreases.

[0067] The visual feedback material handling control device for multi-specification hardware parts provided in the embodiments of the present invention can execute the visual feedback material handling control method for multi-specification hardware parts provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0068] Although this application makes various references to certain modules in the apparatus according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not intended to limit the scope of protection of this invention.

[0069] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A visual feedback pick control method for multi-gauge hardware, characterized in that, The method comprises: acquiring real-time images of the hardware part stacking area by a multi-light source vision unit, generating three-dimensional point cloud data, performing multi-specification hardware part segmentation and identification based on the three-dimensional point cloud data, and determining the specification type information of the to-be-taken parts; performing to-be-taken part analysis based on the specification type information according to the spatial pose of the multi-specification hardware part, generating a to-be-taken part parameter set, and the to-be-taken part parameter set comprising to-be-taken part path parameters and to-be-taken part clamp parameters; performing real-time visual tracking feedback by the to-be-taken part path parameters, and drawing a pose change trend graph of the multi-specification hardware part; performing deviation calculation on the to-be-taken part path parameters based on the pose change trend graph, correcting the to-be-taken part trajectory according to the path deviation amount, and performing control optimization according to the correction result combined with the to-be-taken part clamp parameters, and formulating a to-be-taken part control scheme; wherein, the to-be-taken part analysis based on the specification type information according to the spatial pose of the multi-specification hardware part, and the method comprises: performing to-be-taken part clamp analysis on the to-be-taken part equipment based on the specification type information, and determining clamp angle parameters; performing to-be-taken part motion analysis on the to-be-taken part equipment according to the clamp angle parameters, and determining motion speed parameters; performing to-be-taken part pose analysis according to the spatial pose of the multi-specification hardware part combined with the specification type information, and determining pose tolerance parameters; performing to-be-taken part clamping simulation of the multi-specification hardware part based on the clamp angle parameters, and determining the to-be-taken part clamp parameters; performing to-be-taken part trajectory analysis based on the motion speed parameters combined with the pose tolerance parameters, and determining to-be-taken part path parameters; wherein, the to-be-taken part trajectory analysis based on the motion speed parameters combined with the pose tolerance parameters, and the method for determining to-be-taken part path parameters comprises: performing pose rotation analysis based on the pose tolerance parameters, and constructing a pose matrix; performing to-be-taken part reachability analysis on the pose matrix based on the motion speed parameters, extracting to-be-taken part trajectories when the pose matrix is to-be-taken part reachable, and determining the to-be-taken part path parameters; when the pose matrix is to-be-taken part unreachable, triggering a repositioning instruction, searching the pose matrix through the repositioning instruction, checking the search result by calling the motion speed parameters, updating the pose matrix until the pose matrix is to-be-taken part reachable, and extracting the to-be-taken part path parameters.

2. The visual feedback pick control method for multi-specification hardware of claim 1, wherein, acquiring real-time images of the hardware part stacking area by a multi-light source vision unit, and generating three-dimensional point cloud data, the method comprising: the multi-light source vision unit adopts coaxial light source and ring light source combination lighting, activates binocular stereo camera to perform multi-view synchronous acquisition on the hardware part stacking area, and obtains a multi-view original image set; performing stereo matching based on the multi-view original image set, and generating initial three-dimensional point cloud; performing hardware part metal reflection compensation according to the initial three-dimensional point cloud, and determining photometric stereo fusion data; enhancing the initial three-dimensional point cloud according to the photometric stereo fusion data, and determining the three-dimensional point cloud data.

3. The visual feedback pick control method for multi-specification hardware of claim 1, wherein, performing multi-specification hardware part segmentation and identification based on the three-dimensional point cloud data, and determining the specification type information of the to-be-taken parts, the method comprising: performing geometric analysis on multi-specification hardware parts according to the three-dimensional point cloud data, and generating geometric feature vectors; The contour parameters are determined by contour recognition of the multi-specification hardware based on the geometric feature vector; The multi-specification hardware is clustered and segmented according to the contour parameters, and a plurality of independent hardware structure parameters are obtained; The specification type information of the to-be-taken hardware is determined by specification recognition based on the plurality of independent hardware structure parameters and the three-dimensional point cloud data.

4. The visual feedback pick control method for multi-specification hardware of claim 3, wherein, The specification type information of the to-be-taken hardware is determined by specification recognition based on the plurality of independent hardware structure parameters and the three-dimensional point cloud data, and the method comprises: The three-dimensional point cloud data is arranged according to the plurality of independent hardware structure parameters, and a point cloud normal vector histogram is constructed; Point cloud curvature calculation is performed based on the point cloud normal vector histogram, extreme values are extracted according to the calculation results, and point cloud curvature extreme points are obtained; The point cloud curvature extreme points are taken as seed points of the point cloud normal vector histogram for region growing, and a growing region volume data is obtained; When the growing region volume reaches a preset minimum volume threshold, the region growing of the seed points is terminated, the growing region volume is extracted, the specification classification of the plurality of independent hardware structure parameters is performed, and the specification type information of the to-be-taken hardware is determined.

5. The visual feedback pick control method for multi-specification hardware of claim 1, wherein, Real-time visual tracking feedback is performed on the taking path parameters, and a pose change trend graph of the multi-specification hardware is drawn, and the method comprises: The taking path parameters are executed, the high-speed vision unit is activated to track the taking equipment, and a taking dynamic image sequence of the multi-specification hardware is obtained; Adjacent frame matching is performed on the pose matrix according to the taking dynamic image sequence, and a plurality of pose adjacent frame data is extracted; Displacement calculation is performed based on the plurality of pose adjacent frame data, and an adjacent frame pose change matrix is obtained; Based on the adjacent frame pose change matrix, a trend is identified according to the change time sequence, the identification results are connected adjacent to each other, and the pose change trend graph is drawn.

6. The visual feedback pick control method for multi-specification hardware of claim 1, wherein, Deviation calculation is performed on the taking path parameters based on the pose change trend graph, and the method comprises: Six-degree-of-freedom pose time sequence data is obtained by analyzing the pose change trend graph; Double-branch trajectory judgment is performed according to the taking path parameters based on the six-degree-of-freedom pose time sequence data, a trajectory judgment result is generated, and the trajectory judgment result contains a collision judgment result or a deviation judgment result; When the trajectory judgment result is the collision judgment result, the taking path parameters are intersected according to the six-degree-of-freedom pose time sequence data, a path collision point is determined, and the path collision point is added to the path deviation amount; When the trajectory judgment result is the deviation judgment result, the taking path parameters are progressively identified away according to the six-degree-of-freedom pose time sequence data, deviation progressive data is determined, the taking path parameters are traced back according to the deviation progressive data, and a deviation starting point is determined; The deviation progressive data and the deviation starting point are added to the path deviation amount.

7. The visual feedback pick control method for multi-specification hardware of claim 6, wherein, Double-branch trajectory judgment is performed according to the taking path parameters based on the six-degree-of-freedom pose time sequence data, and a trajectory judgment result is generated, and the method comprises: The taking path parameters are traversed and matched based on the six-degree-of-freedom pose time sequence data, and a pose path matching result is generated. Performing distance continuous analysis based on the pose path matching result to obtain a distance gradual change parameter; When the distance gradual change parameter is gradually increasing, the deviation determination result is generated; When the distance gradual change parameter is gradually decreasing, the collision determination result is generated.

8. A visual feedback material handling control device suitable for multi-specification hardware parts, characterized in that, The device is used to implement the visual feedback control method for multi-specification hardware parts according to any one of claims 1 to 7, and the device comprises: A hardware part segmentation and recognition module is configured to collect real-time images of a hardware part stacking area through a multi-light source vision unit, generate three-dimensional point cloud data, perform multi-specification hardware part segmentation and recognition based on the three-dimensional point cloud data, and determine specification type information of a part to be taken out; A taking-out analysis module is configured to perform taking-out analysis based on the specification type information according to the spatial pose of the multi-specification hardware part, generate a taking-out parameter set, and the taking-out parameter set comprises a taking-out path parameter and a taking-out clamp parameter; A visual tracking feedback module is configured to perform real-time visual tracking feedback according to the taking-out path parameter, and draw a pose change trend graph of the multi-specification hardware part; A taking-out path deviation calculation module is configured to perform deviation calculation on the taking-out path parameter based on the pose change trend graph, perform taking-out trajectory correction according to the path deviation amount, perform control optimization according to the correction result and the taking-out clamp parameter, and formulate a taking-out control scheme.

Images