Intelligent manufacturing feeding method and system integrating three-dimensional vision and robot cooperation

Abstract

The invention discloses an intelligent manufacturing feeding method and system fusing three-dimensional vision and robot cooperation, and belongs to the technical field of robot automation control. Three-dimensional point cloud data in a feeding area is acquired, and a workpiece space topological relation graph is constructed through adaptive clustering and boundary extraction; generating a task association tensor in combination with the physical attribute of the workpiece and a scheduling rule; the method comprises the following steps: collecting operation state information of a plurality of industrial robots, and constructing an action capability tensor; based on the task association tensor and the action capability tensor, an optimal task allocation matrix is generated through a multi-target collaborative optimization algorithm, and dynamic matching between the robot and the workpiece is achieved; in combination with real-time point cloud feedback, the initial path is corrected and the tail end is adjusted, and the robot is driven to accurately execute a grabbing task; the intelligent feeding system has the advantages of being high in environment adaptability, high in dispatching intelligence degree, excellent in path control precision and the like, and is suitable for high-dynamic complex manufacturing scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot automation control, in particular to an intelligent manufacturing feeding method and system fusing three-dimensional vision and robot cooperation. BACKGROUND

[0002] In the automatic production line, intelligent feeding of complex materials is always one of the important bottlenecks restricting the efficiency and flexibility of the whole process automation. The existing feeding system mainly relies on the way of regular placement and fixed-point grabbing to realize material transmission, which needs to cooperate with fixed molds, sensors or manual intervention to ensure the feeding accuracy. This way has poor adaptability and low efficiency when facing the manufacturing scene of irregular workpiece stacking, various types or frequent batch switching.

[0003] Although some systems try to introduce three-dimensional vision technology to assist in identifying the position and attitude of the workpiece, most of them are isolated operation models of "perception-execution", which cannot realize the integrated logic of perception-decision-cooperation. The three-dimensional vision data is not efficiently utilized for collaborative strategy planning, resulting in high recognition accuracy but low overall scheduling efficiency. In addition, the existing systems generally rely on preset task scheduling rules and lack dynamic perception and intelligent cooperation of the resource state among robots, which cannot adaptively adjust task allocation or path optimization in a multi-workpiece and dynamic task distribution environment, restricting their application in high variability manufacturing scenarios.

[0004] More importantly, the existing methods do not fuse the deep coupling logic between the three-dimensional point cloud space information and the robot action space model, resulting in path planning that is disconnected from the physical characteristics of the workpiece, making it difficult to realize scene-level perception-cognition-execution closed-loop control.

[0005] Therefore, there is an urgent need for a technical solution that can construct a workpiece space feature map based on three-dimensional vision and fuse the current state and constraint conditions of multiple robots to dynamically generate an optimal collaborative grabbing strategy, forming an integrated intelligent feeding method of "heterogeneous perception-multi-agent cooperation-path closed-loop correction", which fundamentally breaks through the limitations of isolated identification and fixed scheduling of existing technologies, and realizes the adaptive evolution and efficient execution of the manufacturing process. SUMMARY

[0006] The purpose of the present application is to provide an intelligent manufacturing feeding method and system fusing three-dimensional vision and robot cooperation to solve the problems in the background art.

[0007] In order to achieve the above purpose, the present application provides the following technical solution: an intelligent manufacturing feeding method fusing three-dimensional vision and robot cooperation, comprising: S100, acquiring point cloud data of a material stacking area in a feeding area, and constructing a workpiece space topology graph through a spatial feature clustering algorithm, including three-dimensional attitude parameters, boundary features and occlusion relationships of each workpiece; S200, generating a task association tensor based on the topological relationship graph, combining the physical properties of the workpieces and the task priority rules, for expressing the comprehensive matching degree of the workpieces in the spatial relationship, processing priority and grasping feasibility dimensions; S300, collecting current motion state information of the plurality of collaborative industrial robots, and constructing an action capability tensor including robot reachable space, end load state, tool type and operation time delay; S400, generating an optimal task allocation matrix based on the task association tensor and the action capability tensor through a multi-objective collaborative optimization algorithm, realizing dynamic matching between the workpieces and the robots, and the optimization algorithm includes a target function constraint through path time consumption, collision avoidance and overall efficiency maximization; S500, mapping the allocation matrix to a specific grasping path of the robot, combining real-time point cloud feedback information for path correction and end adjustment, and driving the matched robot to perform the feeding operation; S600, if a new workpiece enters the area or the existing workpiece posture changes during the feeding process, automatically recalculating the graph and the tensor, and iteratively updating steps S200 to S500.

[0008] Preferably, the S100 comprises: S101, collecting high-density point cloud data in the feeding area, and using an adaptive clustering algorithm based on density peaks to pre-segment the point cloud, and removing background noise points and overlapping areas; S102, constructing a local surface normal vector field based on the clustering results, extracting the boundary contour of each workpiece combining the Gaussian curvature and edge gradient information, and estimating the three-dimensional posture parameters of each workpiece including pitch angle, yaw angle and roll angle using a posture fitting model; S103, constructing an occlusion dependency graph based on the depth continuity of the overlapping area of the workpiece boundary bounding box, predicting the occlusion order and grasping feasibility score using a graph neural network, and finally forming a workpiece spatial topological relationship graph.

[0009] Preferably, the S200 comprises: S201, extracting the posture stability, center of gravity offset and boundary integrity of each workpiece in the topological relationship graph as a physical property feature vector, and constructing a physical property matrix P; S202, setting a task priority factor set Q according to the process scheduling requirement, workpiece processing batch identification and occlusion level in the topological graph, and mapping the priority in different dimensions to a unified scale space through a normalization function; S203, combining the physical property matrix P and the priority factor set Q to generate a task association tensor T in the form of a three-dimensional tensor, and each element Tij in T represents the weighted matching degree of workpiece i and workpiece j in grasping feasibility, spatial independence and processing priority.

[0010] Preferably, the S300 comprises: S301, based on real-time joint encoders and forward kinematics, calculate the dynamic reachable space area of each robot at the current time, and generate a three-dimensional reachability grid map; S302, synchronously acquire the load state, clamp type parameters and historical average response time delay of each robot end effector, and combine them in a standardized manner to form a state feature vector; S303, all robot state feature vectors are spliced according to task response dimension, tool constraint dimension and space topology dimension to construct an action capability tensor Rijk, which is used to express the scheduling capability score of robot i under task j and tool configuration k.

[0011] Preferably, the S400 comprises: S401, input the task association tensor and the action capability tensor into the heterogeneous graph matching module, construct a weighted bidirectional graph model between the workpiece nodes and the robot nodes, and the edge weight is obtained by joint tensor calculation of T and R; S402, design a multi-objective optimization function containing path time consumption C, collision probability P and task completion rate E ; In the formula, h1, h2, h3 are weight factors of three targets respectively, satisfying h1+h2+h3=1; S403, adopt a genetic optimization algorithm based on local disturbance and gradient reinforcement to iteratively solve the graph model, and output an optimal task allocation matrix Tij representing the optimal matching degree of robot j executing workpiece i task under given constraints.

[0012] Preferably, the S500 comprises: S501, based on the optimal task allocation matrix, call the robot inverse kinematics model and workpiece pose data to generate an initial grasping path; S502, during the robot grasping process, continuously collect target workpiece point cloud flow, and use an ICP-based local point cloud fitting algorithm to track workpiece micro-displacement and attitude drift in real time; S503, generate a path correction vector ΔP according to the point cloud drift result, and update the grasping path in an interpolation manner , for real-time path, while dynamically adjusting the end effector clamping angle and position.

[0013] Preferably, the S600 comprises: S601, set a three-dimensional point cloud change monitoring threshold Δθ, when detecting that a new point cloud cluster is added in the feeding area or the attitude deviation of the existing workpiece exceeds Δθ, trigger the asynchronous reconstruction mechanism; S602, call the incremental update module to locally reconstruct the constructed workpiece space topology graph, and only execute space clustering and boundary fitting on the newly added or posture mutated workpiece nodes; S603, synchronize the incremental update task associated tensor and the action ability tensor, and re-execute the task allocation and path re-planning under the current robot state, iteratively trigger steps S200 to S500, and ensure that the system continuously maintains the optimal grasping collaborative state under dynamic working conditions.

[0014] The application provides an intelligent manufacturing feeding system integrating three-dimensional vision and robot collaboration, comprising: A three-dimensional vision perception module acquires point cloud data of a material stacking area in a feeding area, and constructs a workpiece space topology graph through a spatial feature clustering algorithm, including three-dimensional posture parameters, boundary features and occlusion relationships of each workpiece. A task modeling and matching module generates a task associated tensor based on the topology graph, combines workpiece physical properties and task priority rules, and generates a task associated tensor for expressing the comprehensive matching degree of workpieces in the spatial relationship, processing priority and grasping feasibility dimensions. A robot state modeling module collects current motion state information of multiple collaborative industrial robots, and constructs an action ability tensor, including robot reachable space, end load state, tool type and operation time delay. A scheduling optimization and task allocation module generates an optimal task allocation matrix based on the task associated tensor and the action ability tensor through a multi-objective collaborative optimization algorithm, realizes dynamic matching between workpieces and robots, and the optimization algorithm includes a target function constraint through path time consumption, collision avoidance and overall efficiency maximization. A path generation and execution control module maps the allocation matrix into a specific grasping path of the robot, combines real-time point cloud feedback information for path correction and end adjustment, and drives the matched robot to perform the feeding operation. An adaptive module automatically recalculates the graph and tensor and iteratively updates them if new workpieces enter the area or the posture of existing workpieces changes during the feeding process.

[0015] In the above technical solution, the application provides technical effects and advantages: 1. The application realizes a flexible feeding method for complex manufacturing scenarios by integrating three-dimensional vision perception, task modeling, robot capability modeling and multi-objective scheduling optimization. The scheme introduces point cloud clustering, topology graph construction, tensor modeling and graph matching algorithms, effectively solving the problems of inaccurate identification of dynamic stacked workpieces, static rigidity of scheduling strategies and lack of closed-loop correction of paths in the prior art. Experimental results show that the application has significant advantages in improving recognition accuracy, path stability and task response speed.

[0016] 2、The application enables the robot to update the decision logic in real time and iteratively optimize the execution path when facing workpiece changes, task disturbances or multi-target conflicts, thereby realizing the technical leap from "static rule execution" to "dynamic intelligent scheduling". The overall scheme has high modularity and algorithm-driven characteristics, improving the intelligent level and environmental adaptability of the industrial system, and is suitable for various complex and high-variability automated feeding scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0018] Figure 1 Mind map for the method of the present application.

[0019] Figure 2 Mind map for the system module of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] Embodiment 1, please refer to Figure 1 The intelligent manufacturing feeding method described in this embodiment fuses three-dimensional vision and robot cooperation, which includes: S100, acquiring point cloud data of a material stacking area in a feeding area, and constructing a workpiece space topology graph through a spatial feature clustering algorithm, including three-dimensional pose parameters, boundary features and occlusion relationships of each workpiece; S200, based on the topology graph, combining workpiece physical properties and task priority rules, generating a task association tensor for expressing the comprehensive matching degree of workpieces in the spatial relationship, processing priority and grasping feasibility dimensions; S300, collecting current motion state information of a plurality of collaborative industrial robots, and constructing an action capability tensor, including robot reachable space, end load state, tool type and operation time delay; S400, based on the task correlation tensor and the action ability tensor, an optimal task allocation matrix is generated through a multi-objective collaborative optimization algorithm to realize dynamic matching between the workpieces and the robots, and the optimization algorithm includes constraints of objective functions through path time consumption, collision avoidance and overall efficiency maximization; S500, the allocation matrix is mapped to a specific grasping path of the robot, path correction and end adjustment are performed in combination with real-time point cloud feedback information, and the matched robot is driven to perform the feeding operation; S600, if a new workpiece enters the area or the posture of the existing workpiece changes during the feeding process, the atlas and the tensor are automatically recalculated, and steps S200 to S500 are iteratively updated.

[0022] In the intelligent manufacturing feeding method fusing three-dimensional vision and robot collaboration described in the application, first, the workpieces in the feeding area are identified and modeled in three dimensions through the S100 step, so as to provide accurate and structured input information for subsequent task scheduling and path planning. This step includes point cloud data acquisition, spatial clustering segmentation, boundary posture extraction and occlusion relationship modeling, which are as follows: First, a structured light three-dimensional vision sensor (such as a multi-line laser scanner or an industrial-grade RGB-D camera) deployed above the feeding station is used to scan the entire feeding area, and high-density point cloud data containing the workpieces to be grasped are obtained. In order to ensure the accuracy of subsequent identification, the point cloud resolution is preferably set to be less than 1 mm, and the sampling frame rate is kept above 30 Hz to capture dynamic changes.

[0023] After obtaining the original point cloud data, an adaptive clustering algorithm based on density peaks is used to pre-segment the point cloud data. After calculating the local density p and the distance d of each point from the high-density points, the clustering center is automatically determined through the density decision diagram, and adaptive identification of workpieces of different scales and arrangements is realized. Compared with traditional K-means or DBSCAN algorithms, this method does not depend on manual specification of the number of clusters, and has better generalization and stability.

[0024] Based on clustering, conditional filtering and radius threshold method are further used to remove background noise points, misidentified points and overlapping interference regions adjacent to the boundary to improve the purity and clarity of workpiece boundary extraction.

[0025] After pre-segmentation, based on each independent workpiece point cloud subset, its local surface normal vector field is constructed. Specifically, the principal direction vector of each point in its neighborhood window (generally with a radius of 5-10 mm) is calculated through principal component analysis (PCA), and is fitted as a local tangent plane, so that the unit normal vector n of each point is solved.

[0026] Subsequently, the Gaussian curvature K and the edge gradient information G are calculated in combination with the normal vector change rate to identify each workpiece contour boundary. The boundary extraction method uses an improved edge curvature threshold method to automatically identify boundary mutation regions and the boundary lines of adjacent workpieces, effectively improving the boundary integrity and recognition accuracy.

[0027] On the basis of extracting the contour, a workpiece pose fitting model is constructed. The model uses the least squares method to fit the major axis direction of the workpiece bounding box in length, width and height, and calculates the Euler angle thereof relative to the three-dimensional coordinate system to obtain a complete set of pose parameters including the pitch angle (Pitch), yaw angle (Yaw) and roll angle (Roll). The pose estimation method can adapt to irregularly shaped workpieces and improve the accuracy of the end pose adjustment of the robot.

[0028] After the boundary and pose information of each workpiece is extracted, the occlusion relationship modeling stage is entered. First, an initial occlusion possibility graph is constructed based on the spatial overlap region of the workpiece bounding box. By analyzing the depth continuity in the intersection region between two workpieces, it is determined whether there is an occlusion: if the depth change in the region is less than a set threshold Δd (such as 5 mm), it is considered that there is an occlusion contact.

[0029] Subsequently, all workpieces are regarded as nodes in a graph neural network (GNN), and the occlusion edges between the workpieces are regarded as directed edges, which are input into the graph neural network for occlusion order prediction. The network structure uses a 3-layer GCN (Graph Convolutional Network) encoder, which combines node pose, boundary area and relative position as feature input, and outputs the occlusion level score D and the grasping feasibility score F of each node.

[0030] Finally, based on the above scores, a workpiece spatial topology graph G is constructed. The graph G not only contains the position coordinates, three-dimensional pose and boundary contour features of each workpiece, but also integrates the occlusion level relationship and grasping priority between workpieces, providing multi-dimensional and structured data support for subsequent task-related tensor generation and robot action capability matching.

[0031] This step connects the path between point cloud low-level perception and high-order spatial semantic modeling through density peak adaptive clustering, local curvature field fitting and graph neural network occlusion reasoning, not only improves the stability and boundary accuracy of workpiece recognition, but also for the first time introduces a topology graph form to express the spatial scene relationship in industrial vision, realizes the graspability reasoning and pose priority sorting between workpieces.

[0032] Compared with the traditional 2D recognition method relying on rule image contour, the method has stronger robustness and universality, is suitable for stacked disorder, and various dynamic manufacturing environment, and is a key basic step for realizing intelligent manufacturing flexible feeding.

[0033] In the present application, in order to realize intelligent grasping distribution of multiple target workpieces by the robot, a decision structure capable of quantifying the priority of the workpiece and the feasibility of grasping needs to be constructed. To this end, after completing the construction of the workpiece spatial topological relation graph, the step S200 is further executed, that is, the task correlation tensor T is constructed. This step comprehensively considers the multi-dimensional information of the physical properties, processing requirements, shielding level, etc. of the workpieces, and completes the matching modeling of the workpieces under the graspability and task priority. Specifically, the following sub-steps are included: Firstly, the structure and posture parameters of each workpiece are extracted from the topological relation graph G, which are used to construct the physical property feature vector and form the physical property matrix P. Specifically: Posture stability S: according to the included angle difference between the current posture angle (pitch, yaw, roll) of the workpiece and the standard grasping posture, if the posture deviation is large, the stability score of the workpiece under the current posture is low; Center of gravity offset C: the offset amplitude of the workpiece center of gravity relative to the envelope center is estimated by point cloud quality distribution, which reflects the influence of gravity and the risk of overturning in the grasping process; Boundary integrity B: the recognition integrity of the workpiece is evaluated by the boundary closure rate and the edge smoothness, and the higher the integrity, the easier the path planning and end clamping control.

[0034] The above three parameters are combined in the form of a vector to form the physical property feature vector F of each workpiece [S, C, B], and then all the F of the workpieces are summarized to construct the physical property matrix P Where n is the number of workpieces to be grasped, and R is the real number set.

[0035] Then, the task priority factor set Q is constructed in combination with the processing task scheduling strategy and the actual process requirement. In order to quantify the multi-dimensional influence of the priority, the following three key dimensions are considered: Processing batch identification L: each workpiece is attached with a processing batch ID, which can be read by the visual recognition system through the two-dimensional code or color coding recognition on the surface of the workpiece. The system will preferentially arrange the workpieces corresponding to the current processing batch; Task scheduling urgency D: according to the processing sequence, delivery cycle and other scheduling parameters, a time sensitivity coefficient is set for each workpiece, and the higher the urgency of the task, the higher the priority; Shielding level Z: the shielding level of each workpiece is extracted from the topological relation graph. The workpiece with high shielding degree has greater grasping difficulty, and the priority is appropriately reduced to reduce path conflicts.

[0036] The three parameters are combined into a priority raw factor vector U=[L,D,Z], and are mapped to the interval [0,1] by a normalization function f(·) to make different dimensions comparable, to obtain a normalized priority factor set Q={q1,q2,...,qn}, wherein qn is the comprehensive priority score of the nth workpiece.

[0037] After obtaining the physical attribute matrix P and the priority factor set Q, the generation phase of the task association tensor T is entered. The task association tensor T is used to describe the spatial independence, grasping feasibility and priority conflict degree of any two workpieces when performing scheduling, and is constructed in the form of T∈ , wherein Tijk represents the matching score of workpiece i and workpiece j in the kth dimension, and the specific calculation method is as follows: The 1st dimension (grasping feasibility): according to the posture stability and boundary integrity scores of the workpieces in the P matrix, the difference in grasping complexity is calculated, and whether it is suitable to be executed in the same batch is obtained; The 2nd dimension (spatial independence): the spatial interference probability of the two workpieces is calculated according to the overlapping degree of the boundary envelope boxes in the topological relationship graph, and the higher the spatial independence, the higher the score; The 3rd dimension (task priority): the absolute value of the difference between the priority scores of the workpieces in Q is used as a conflict index, and the smaller the difference, the closer the task urgency, and the more suitable it is as a task in the same group.

[0038] In order to enhance the expression ability of the tensor T, a weighted fusion strategy is adopted to construct the final score, that is: ; wherein, is the matching score function of workpiece i and j in the grasping feasibility dimension; is the spatial independence score function; is the task priority proximity function; alpha, beta and gamma are the weighting coefficients of the above three dimensions, satisfying alpha+beta+gamma=1, which can be dynamically set according to the actual scheduling scene, and the typical setting is alpha=0.4, beta=0.3, gamma=0.3.

[0039] The finally constructed task association tensor T can provide structured input for subsequent robot task scheduling algorithms, support multi-objective scheduling modeling and graph matching optimization operations.

[0040] In order to realize multi-robot collaborative grasping scheduling, the application further introduces an action capability modeling mechanism on the basis of the construction of the task association tensor, to construct a structured tensor data structure reflecting the current schedulable capability of the robot. This step is S300, including modeling of spatial accessibility of the robot, extraction of end state features and expression of tensor-level task capability, to construct an action capability tensor R, which provides capability boundary constraints for the scheduling algorithm. Specifically as follows: Firstly, for each collaborative robot deployed in the production line, the current joint angles and velocity parameters are obtained based on the real-time joint encoder feedback. Through the Forward Kinematics (FK) model, the workspace region that the end effector can reach under the current joint configuration is calculated. The forward kinematics model is constructed based on the Denavit-Hartenberg parameter table (D-H parameter), and the input is the joint angle at the current time , and the output is the position and attitude vector of the end in three-dimensional space.

[0041] To quantify the scheduling flexibility of the robot in space, a three-dimensional reachability grid map is further constructed based on the reachable point set within multiple time steps, and the grid resolution is generally taken as 5-10 mm. Each grid cell contains the frequency density p of the region being visited, which serves as the spatial scoring basis for robot flexibility. In addition, by analyzing the range of clamping direction change within the reachable grid, the attitude switching ability of the end tool is evaluated.

[0042] While constructing the spatial reachability information, the key operating state indicators of each robot end effector are obtained in real time, including: Load state L: the load value currently carried by the clamp, obtained through force / torque sensors, reflecting whether it is in a near-saturated or redundant state; Clamp type G: records the type of the current robot end clamp (such as vacuum chuck, double-finger gripper, etc.), and maps it with the workpiece adaptation model; Average response delay τ: based on the robot control system task queue record, the completion period of the scheduling task is calculated for a certain number of times, representing the execution efficiency of the robot under the current working condition.

[0043] The above three-dimensional parameters are normalized to generate the state feature vector S of each robot, where I, g, and τ are the normalized load state, clamp adaptation factor, and unit time delay score of the ith robot, respectively.

[0044] After completing the construction of the spatial grid map and the generation of the state vector, the system fuses the spatial capabilities and state features of all robots to construct the action capability tensor R, with the tensor dimension being R∈ , where: m represents the number of robots; n represents the number of workpieces to be grasped; k represents the number of end clamp configuration types or grasping strategy categories.

[0045] Each element Rijk of the action capability tensor R represents the capability score of the ith robot when facing the jth task (i.e., grasping the jth workpiece) and configuring the kth type of clamp or grasping strategy. The scoring mechanism integrates the following three dimensions: Spatial coverage σ: evaluated according to whether the spatial position of task workpiece j is in the reachability grid of robot i; Tool matching degree μ: determined by the degree of adaptation between the fixture type k and the workpiece grasping characteristics (such as surface material, posture constraint); Execution response capability ε: combined with response time delay τ and load state I to evaluate the current scheduling timeliness and execution risk.

[0046] The final score is generated using the following weighting function: ; wherein: are the weighting coefficients of the spatial reachability, tool matching degree and execution efficiency dimensions, and the typical values are set to = 0.4, = 0.3, = 0.3; τ is normalized to the interval [0, 1], and the smaller the scheduling time delay, the higher the score.

[0047] Through the execution of this step, the system establishes a complete modeling chain from the physical execution state of the robot to the task adaptation capability, providing a solid data foundation for subsequent task scheduling optimization and path planning.

[0048] After the task association tensor T and the action capability tensor R are constructed, in order to realize the optimal task scheduling matching relationship between the robot and the workpiece, the present application further executes the S400 step, designs a heterogeneous graph matching model and a multi-objective optimization function, and solves the optimal task allocation matrix M through a genetic optimization algorithm based on local disturbance and gradient reinforcement. The specific steps are as follows: First, the task association tensor T and the action capability tensor R are taken as inputs to construct a robot-workpiece heterogeneous graph model. This graph model includes two types of nodes: Workpiece node Vw: each node corresponds to a workpiece to be grasped; Robot node Vr: each node corresponds to a collaborative robot.

[0049] In the robot-workpiece heterogeneous graph model, the edge weight wij of each edge eij connecting robot j and workpiece i is calculated jointly from the tensor T and the tensor R: ; wherein: denotes the average matching degree of workpiece i and workpiece j extracted from the task association tensor T; φR(i,j) denotes the maximum scheduling score of robot j for workpiece i under all fixture configurations; is a weighting coefficient, generally set to = 0.5, =0.5, and can be dynamically adjusted according to the task allocation tendency. The constructed G is a weighted bidirectional two-dimensional graph, and the edge weight represents the scheduling adaptation degree between the robot and the workpiece. The graph structure supports parallel optimization and dynamic scheduling update.

[0050] After the graph matching model is constructed, in order to achieve the optimization of scheduling target, the following multi-objective scheduling optimization function F is defined to comprehensively evaluate the pros and cons of the task allocation scheme: ; wherein: C is the total time consumption of the task path, including the robot empty moving time, clamping adjustment time and workpiece grabbing completion time; P is the collision probability between robots, which is modeled based on the path trajectory overlap degree and time sequence conflict probability; E is the system task completion rate, which measures the proportion of high priority workpieces processed in time in the allocation scheme; h1, h2 and h3 are weight factors of the three objectives, satisfying h1+h2+h3=1, and the system can dynamically adjust according to the production rhythm strategy, workpiece processing priority or safety requirement, such as setting h1=0.3, h2=0.2 and h3=0.5 to represent more attention to task completion efficiency.

[0051] The above optimization objectives not only cover key performance indicators such as time, path conflict and scheduling effect, but also support strategy fine-tuning according to actual manufacturing requirements, enhancing the scene adaptability of the scheduling algorithm.

[0052] In order to solve the above optimization objective function, an improved genetic optimization algorithm is used, which combines local disturbance mechanism and gradient reinforcement mechanism. The algorithm flow is as follows: Initialize population: generate several groups of task allocation schemes Each scheme is an m×n binary allocation matrix, and the matrix element Mij∈{0,1} represents whether the task of workpiece i is executed by robot j; Fitness evaluation: for each allocation scheme, calculate its corresponding optimization function value F(M) as the individual fitness; Local disturbance operator: for schemes with high collision probability or path congestion, introduce a local repair disturbance mechanism based on tensor R, and preferentially allocate to robots with higher capability scores; Gradient reinforcement mechanism: in the fitness growth stage, the information of the simulated gradient direction is introduced to strengthen the search ability of the current optimal individual and shorten the convergence period; Selection and crossover: use tournament selection and multi-point crossover operation to generate new generation of candidate schemes; Termination condition: output the optimal allocation matrix M* when the set number of generations or the fitness convergence threshold is reached.

[0053] The final optimal task allocation matrix Mij represents the optimal matching result of robot j performing the task of workpiece i under the current space-time and tool constraints. The matrix structure has the following advantages: supporting multi-robot, multi-workpiece, and multi-tool type scheduling adaptation; having structural stability and dynamic updating capability, which can adapt to real-time feeding changes; and being able to directly interface with the path planning module to achieve end-to-end task-driven scheduling execution.

[0054] After completing the construction of the optimal task allocation matrix, to ensure that the robot accurately performs the workpiece grasping task, the application further performs step S500, that is, generates a robot grasping path according to the allocation result, and combines real-time point cloud data to perform path fine-tuning and end dynamic adjustment, thereby achieving high-precision grasping. Specifically, it includes three major links of path initialization, point cloud drift tracking, and path correction, and the technical process is as follows: First, the system determines the target workpiece i to be executed by robot j according to the optimal task allocation matrix Mij generated above. Combined with the three-dimensional pose data of the workpiece (including position coordinates and pitch angle, yaw angle, and roll angle) and the current end state of the robot, the inverse kinematics (IK) model in the robot controller is called to calculate the end grasping pose path from the current position to the target workpiece.

[0055] The inverse kinematics model solves a set of joint angle solutions that satisfy Tdes based on the known end target pose Tdes=[x, y, z, φ, θ, ψ] and the robot structure parameters , and outputs a multi-segment continuous grasping path, including: Approach path: the end tool gradually approaches the workpiece boundary from the idle position; Pose alignment path: the end pose gradually aligns with the target clamping surface; Grasping path: the end gripper performs a precise closing action to complete the grasping.

[0056] The path planning considers factors such as space obstacle avoidance constraints, gripper opening and closing restrictions, and pose accessibility to ensure path physical feasibility and clamping stability.

[0057] During the actual execution of the robot grasping process, to prevent path deviation caused by slight position deviation or pose change of the workpiece, the application introduces a path real-time correction mechanism based on point cloud fitting. This mechanism continuously collects the point cloud stream of the target workpiece at a high frequency (recommended not less than 20 Hz), and combines the reference point cloud of the last frame for real-time alignment.

[0058] The Iterative Closest Point (ICP) algorithm is used to fit the current point cloud Pt of the target workpiece with the initial point cloud P0, and calculate the optimal rigid transformation matrix TΔ, which is composed of a three-dimensional translation vector Δx and a rotation vector Δθ: Δx=[Δx,Δy,Δz] represents the spatial position of the workpiece during execution of the process; Δθ=[Δφ,Δθ,Δψ] represents the micro-rotation angle of the workpiece pose around each axis.

[0059] The ICP algorithm minimizes the Euclidean distance error function between the point clouds: The transformation matrix TΔ represents the spatial drift between the target workpiece and the initial state, which is used to adjust the robot end trajectory in real time.

[0060] According to the pose drift result obtained by ICP, the system constructs a path correction vector ΔP, which represents the interpolation correction amount from the original path P to the real-time path DP: , The real-time path By interpolating ΔP in the current path segment, the path continuity and the smoothness of the end pose transition are realized, and the updated grasping path is finally output: In addition, in order to further improve the clamping stability, the system dynamically adjusts the end effector pose based on the change trend of Δθ, including: Clamping angle adjustment: fine-tune the docking direction of the clamp according to the target surface normal vector to avoid clamping failure caused by angle deviation; Position compensation: combine Δx data to perform position fine-tuning on the TCP (Tool Center Point) to ensure close fitting with the target surface; Attitude buffer control: if Δθ appears nonlinear mutation, automatically switch to deceleration attitude entering strategy to improve system stability.

[0061] This path updating mechanism replaces full path re-planning with small-scale dynamic interpolation, has the characteristics of high computational efficiency, strong stability, good real-time performance, etc., and is especially suitable for application scenarios such as micro-disturbance in the grasping stage, workpiece not fixed, or dynamic flow line.

[0062] In summary, the present application realizes the whole-process closed-loop grasping path optimization control through the path initialization mechanism based on the optimal task allocation matrix, the real-time point cloud feedback correction algorithm combined with ICP fitting, and the dynamic adjustment strategy of the end clamping pose, and provides a highly adaptive and grasping precision technical solution for intelligent manufacturing feeding task.

[0063] In order to improve the adaptability of the system to dynamic manufacturing environment, the present application further designs a monitoring and response mechanism for dynamic working conditions based on the basic task allocation and path execution mechanism, which is specifically embodied in the execution of S600 step. This step constructs a control closed loop around key processes such as point cloud change monitoring, local incremental update of atlas and tensor, and task scheduling recalculation, so that the system has the ability of adaptive perception, incremental decision and automatic repair.

[0064] Firstly, the system continuously monitors the three-dimensional point cloud changes of the loading area. Through the inter-frame difference analysis of the point cloud sequence collected periodically (e.g. every 100 ms), it is determined whether there are events such as new workpieces, pose mutations or occlusion changes.

[0065] Specifically, a pose change monitoring threshold Δθ is set to quantify the degree of change of the pose of the workpiece between the current frame and the reference frame. The three-axis pose difference of the i-th workpiece at the current time is defined as: When Δθi exceeds the preset threshold (e.g. set to 5°), or the boundary size of the newly added point cloud cluster exceeds the minimum workpiece size standard, the asynchronous reconstruction mechanism is triggered.

[0066] This mechanism ensures that the system has real-time response capability to the following situations: New workpieces automatically enter the loading area; Existing workpieces have significant changes in pose due to vibration, manual intervention or pipeline movement; The occlusion relationship reoccurs overlap or decoupling.

[0067] After triggering the reconstruction mechanism, the system calls the incremental update module to locally reconstruct the original workpiece spatial topology graph G to minimize irrelevant area reconstruction and improve update efficiency.

[0068] The specific process includes: Identify the workpiece node set G' that needs to be reconstructed: the system detects the pose change and identifies new targets, and only selects the point cloud subsets of the workpieces that are newly added or have pose mutations; Local clustering reconstruction: for each point cloud subset in G', the adaptive clustering based on density peaks is re-executed to ensure the continuity of boundary extraction; Boundary fitting and pose estimation: the boundary envelope box, surface normal vector and pose Euler angle of each target are recalculated, and the information of the corresponding nodes in the graph is updated; Local reconnection of occlusion relationship: according to the new boundary position, the occlusion edges between the node and other workpieces are re-established to form the corrected graph structure G*.

[0069] Since only part of the nodes perform the above operations, this incremental update strategy can save more than 70% of the processing time compared to traditional full graph reconstruction, and is suitable for application in production scenarios with high real-time requirements.

[0070] After the graph G* is updated, the system synchronously triggers the incremental reconstruction of the task association tensor T and the action capability tensor R.

[0071] For the task association tensor T, only the task matching items involving the workpieces in G' are recalculated; For the action capability tensor R, only the scheduling score item of the robot and the workpiece in G' is recalculated; The unchanged part of the tensor data is reserved, and the data utilization efficiency is improved.

[0072] The updated tensor is input to the scheduling module, and the system re-executes the whole process of S200 to S500 based on the existing robot state, including: Task priority update; Recalculating the optimal task allocation matrix M; Path generation and execution instruction reissue.

[0073] If there is an existing task being executed, the system will adopt a two-stage scheduling strategy in combination with the current grabbing progress: The task being executed remains unchanged; The remaining tasks are re-allocated according to the new graph to avoid interrupting the execution operation.

[0074] Through the implementation of this step, the present application not only realizes efficient collaborative grabbing in a static scenario, but also has the ability to automatically adapt to changes, independently schedule corrections, and continuously optimize execution in a dynamic environment, significantly improving the intelligence and industrial applicability of the robot collaborative feeding system.

[0075] Embodiment 2, please refer to Figure 2 The intelligent manufacturing feeding system described in this embodiment fuses three-dimensional vision and robot collaboration, including: A three-dimensional vision perception module acquires point cloud data of a material stacking area in a feeding area, and constructs a workpiece spatial topology graph through a spatial feature clustering algorithm, including three-dimensional pose parameters, boundary features and occlusion relationships of each workpiece; A task modeling and matching module generates a task association tensor based on the topology graph, in combination with workpiece physical properties and task priority rules, for expressing the comprehensive matching degree of workpieces in the spatial relationship, processing priority and grabbing feasibility dimensions; A robot state modeling module collects current motion state information of multiple collaborative industrial robots, and constructs an action capability tensor, including robot reachable space, end-of-arm load state, tool type and operation time delay; A scheduling optimization and task allocation module generates an optimal task allocation matrix based on the task association tensor and the action capability tensor through a multi-objective collaborative optimization algorithm, to realize dynamic matching between workpieces and robots, and the optimization algorithm includes a target function constraint through path time consumption, collision avoidance and overall efficiency maximization; A path generation and execution control module maps the allocation matrix to a specific grabbing path of the robot, combines real-time point cloud feedback information for path correction and end adjustment, and drives the matched robot to perform a feeding operation; Adaptive module, if the new workpiece enters the area or the existing workpiece posture changes in the feeding process, the atlas and tensor are automatically recalculated and iteratively updated.

[0076] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A smart manufacturing material feeding method integrating 3D vision and robot collaboration, characterized by: include: S100: Obtain point cloud data of the material stacking area within the loading area, and construct a workpiece spatial topology map using a spatial feature clustering algorithm, including the three-dimensional pose parameters, boundary features, and occlusion relationships of each workpiece. S200. Based on the topological relationship graph, combined with the physical properties of the workpiece and the task priority rules, a task association tensor is generated to express the comprehensive matching degree between workpieces in terms of spatial relationship, processing priority and grasping feasibility. S300 collects current motion state information of multiple collaborative industrial robots and constructs motion capability tensors, including robot reachable space, end-effector load status, tool type and operation latency; S400. Based on the task association tensor and motion capability tensor, an optimal task allocation matrix is ​​generated through a multi-objective collaborative optimization algorithm to achieve dynamic matching between workpieces and robots. The optimization algorithm includes objective function constraints that maximize path consumption, collision avoidance, and overall efficiency. S500 maps the allocation matrix to the robot’s specific grasping path, and combines real-time point cloud feedback information to correct the path and adjust the end effector, driving the matched robot to perform the loading operation. S600. If a new workpiece is detected entering the area or the posture of an existing workpiece changes during the feeding process, the graph and tensor are automatically recalculated, and steps S200 to S500 are iteratively updated.

2. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: S100 includes: S101. Collect high-density point cloud data in the feeding area and use an adaptive clustering algorithm based on density peak to pre-segment the point cloud, removing background noise and overlapping areas. S102. Based on the clustering results, construct a local surface normal vector field, combine Gaussian curvature and edge gradient information to extract the boundary contour of each workpiece, and use an attitude fitting model to estimate the three-dimensional attitude parameters of each workpiece, including pitch angle, yaw angle and roll angle. S103. Based on the depth continuity of the overlapping area of ​​the workpiece boundary bounding box, an occlusion dependency graph is constructed. A graph neural network is used to predict the occlusion order and the feasibility score of grasping, and finally a workpiece spatial topology graph is formed.

3. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: S200 includes: S201. Extract the attitude stability, center of gravity offset and boundary integrity of each workpiece in the topological relationship map as physical attribute feature vectors, and construct the physical attribute matrix P. S202. Set the task priority factor set Q according to the process scheduling requirements, workpiece processing batch identifier and occlusion level in the topology diagram, and map the priorities under different dimensions to a unified scale space through a normalization function. S203. Combining the physical attribute matrix P and the priority factor set Q, a task association tensor T is generated in the form of a three-dimensional tensor. Each element Tij in T represents the weighted matching degree between workpiece i and workpiece j under grasping feasibility, spatial independence and processing priority.

4. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: The S300 includes: S301. Based on real-time joint encoders and forward kinematics calculation, calculate the dynamic reachable space region of each robot at the current moment and generate a three-dimensional reachability mesh map. S302. Synchronously acquire the load status, fixture type parameters and historical average response delay of each robot end effector, and combine them in a standardized manner to form a state feature vector. S303. Perform tensor concatenation on all robot state feature vectors according to the task response dimension, tool constraint dimension, and spatial topology dimension to construct the motion capability tensor Rijk, which is used to express the scheduling capability score of robot i under task j and tool configuration k.

5. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: The S400 includes: S401. Input the task association tensor and motion capability tensor into the heterogeneous graph matching module to construct a weighted bidirectional graph model between workpiece nodes and robot nodes. The edge weights are calculated by the joint tensor of T and R. S402. Design a multi-objective optimization function that includes path time C, collision probability P, and task completion rate E. In the formula, h1, h2, and h3 are the weighting factors of the three objectives, satisfying h1 + h2 + h3 = 1; S403. The graph model is iteratively solved using a genetic optimization algorithm based on local perturbation and gradient reinforcement, and the optimal task assignment matrix Tij is output, which represents the optimal matching degree of robot j in performing task i under given constraints.

6. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: The S500 includes: S501. Based on the optimal task allocation matrix, the robot inverse kinematics model and workpiece posture data are called to generate the initial grasping path; S502. During the robot's grasping process, the point cloud flow of the target workpiece is continuously collected, and the micro-displacement and attitude drift of the workpiece are tracked in real time using the ICP-based local point cloud fitting algorithm. S503. Generate a path correction vector ΔP based on the point cloud drift results, and update the capture path using interpolation. , It provides a real-time path and dynamically adjusts the clamping angle and position of the end effector.

7. The intelligent manufacturing feeding method integrating 3D vision and robot collaboration according to claim 1, characterized in that: The S600 includes: S601. Set the three-dimensional point cloud change monitoring threshold Δθ. When a new point cloud cluster or the attitude deviation of an existing workpiece in the loading area is detected to exceed Δθ, the asynchronous reconstruction mechanism is triggered. S602. Call the incremental update module to perform local reconstruction of the constructed workpiece spatial topology map, and perform spatial clustering and boundary fitting only on newly added or abruptly changed workpiece nodes. S603. Synchronously and incrementally update the task association tensor and motion capability tensor, and re-execute task allocation and path replanning in the current robot state, iteratively triggering steps S200 to S500 to ensure that the system continuously maintains the optimal grasping and coordination state under dynamic working conditions.

8. An intelligent manufacturing feeding system integrating 3D vision and robot collaboration, used to implement the intelligent manufacturing feeding method integrating 3D vision and robot collaboration as described in any one of claims 1-7, characterized in that: include: The 3D vision perception module acquires point cloud data of the material stacking area within the loading area and constructs a workpiece spatial topology map through a spatial feature clustering algorithm, including the 3D pose parameters, boundary features, and occlusion relationships of each workpiece. The task modeling and matching module, based on the topological relationship graph and combined with the physical properties of the workpiece and the task priority rules, generates a task association tensor to express the comprehensive matching degree between workpieces in terms of spatial relationship, processing priority and grasping feasibility. The robot state modeling module collects the current motion state information of multiple collaborative industrial robots and constructs a motion capability tensor, including the robot's reachable space, end-effector load state, tool type, and operation latency. The scheduling optimization and task allocation module generates the optimal task allocation matrix based on the task association tensor and motion capability tensor through a multi-objective collaborative optimization algorithm, thereby realizing dynamic matching between workpieces and robots. The optimization algorithm includes objective function constraints that maximize path consumption, collision avoidance, and overall efficiency. The path generation and execution control module maps the allocation matrix to the robot’s specific grasping path, combines real-time point cloud feedback information to perform path correction and end-effector adjustment, and drives the matched robot to perform the loading operation. The adaptive module automatically recalculates the graph and tensor and iteratively updates them if a new workpiece is detected entering the area or the posture of an existing workpiece changes during the feeding process.

Images