Mechanical arm positioning and grabbing method based on machine vision

Abstract

The invention discloses a mechanical arm positioning and grabbing method based on machine vision, and relates to the technical field of machine vision and mechanical arm control, the method comprises the following steps: synchronously acquiring RGB-D images of a target scene through a multi-view camera array, and generating three-dimensional point cloud data through data fusion; an improved LSD algorithm and a PnP algorithm are adopted to calculate the initial pose of the target object, illumination distortion is eliminated in combination with the generative adversarial network, and three-dimensional coordinates are output; a mechanical arm motion error transfer model is constructed based on Monte Carlo simulation, and a candidate grabbing scheme set is generated through reinforcement learning; and an optimal grabbing scheme is screened through a preset priority evaluation rule, and a mechanical arm joint movement track and a control instruction set are generated. Through multi-modal data fusion and a nonlinear optimization algorithm, the technical problems of large target positioning deviation and sensitive illumination interference in a complex environment are solved, and the grabbing precision and robustness of the mechanical arm are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision and robotic arm control, and particularly relates to a robotic arm positioning and grasping method based on machine vision Background Technique

[0002] With the rapid development of machine vision and industrial automation technologies, robotic arm grasping technology guided by vision has gradually become a key research direction in the field of intelligent manufacturing. Traditional methods usually use monocular or binocular cameras to obtain two-dimensional image information of the target object, achieve object positioning through feature matching and geometric reasoning, and generate the robotic arm motion trajectory based on a preset grasping strategy. For example, in the prior art, RGB-D cameras are generally used to obtain depth information, and the pose parameters are iteratively optimized through point cloud registration and ICP algorithms, and the grasping path is planned by combining inverse kinematics. However, the current traditional methods have significant limitations: in complex lighting environments, the reflection characteristics of the object surface are likely to cause depth data noise and RGB image color distortion, and existing data fusion methods are difficult to effectively suppress the impact of lighting distortion on the accuracy of three-dimensional reconstruction; pose estimation algorithms based on single geometric features (such as SIFT, ORB) are prone to feature matching errors in scenarios where the object is partially occluded or the texture is missing, resulting in the cumulative amplification of pose estimation errors; the robotic arm motion error transfer modeling usually relies on linear approximation or static parameter assumptions, and it is difficult to dynamically adapt to multi-source interferences in actual operations, resulting in insufficient robustness of the grasping scheme. These problems seriously restrict the application effect of robotic arms in industrial sorting, flexible assembly and other scenarios, and there is an urgent need to develop an integrated positioning and grasping solution with high anti-interference ability Summary of the Invention

[0003] Based on this, it is necessary to provide a robotic arm positioning and grasping method based on machine vision that can solve the above problems

[0004] In a first aspect, the present application provides a robotic arm positioning and grasping method based on machine vision, including:

[0005] Synchronously acquire RGB-D images of the target scene using a multi-view camera array, and generate three-dimensional point cloud data through data fusion processing

[0006] Use an improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, and perform lighting distortion elimination processing in combination with a generative adversarial network to generate the three-dimensional coordinates of the target object

[0007] Based on the three-dimensional coordinates, construct a robotic arm motion error transfer model through Monte Carlo simulation, and generate a set of candidate grasping schemes in combination with a reinforcement learning algorithm

[0008] Based on a preset priority evaluation rule, comprehensively score and rank the set of candidate grasping solutions, and select the solution with the highest comprehensive score as the optimal grasping solution;

[0009] According to the parameter characteristics of the optimal grasping solution, generate the joint motion trajectory of the robotic arm and the corresponding control instruction set.

[0010] In one embodiment, the RGB-D image of the target scene is processed by data fusion to generate three-dimensional point cloud data, including:

[0011] Establish the mapping relationship of the RGB-D image of the target scene, use the image registration algorithm to align the RGB-D image, and obtain the aligned image;

[0012] Use the geometric segmentation method based on point cloud normal vector and curvature analysis to segment the target object in the aligned image, and obtain the segmented image;

[0013] Use local binary pattern to extract texture feature of the target object in the segmented image, and obtain the texture feature vector;

[0014] Synchronously encode the geometric boundary point coordinates of the segmented image with the corresponding texture feature vector in space and time, and fuse them to generate a three-dimensional point cloud structured data set containing geometric, texture and timestamp information.

[0015] In one embodiment, use the improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, including:

[0016] Based on the Gaussian kernel function to correct the anisotropic diffusion equation, construct a multi-scale line feature detection model, perform iterative filtering on the three-dimensional point cloud data, and generate a geometric feature point set of the target object contour;

[0017] According to the geometric feature point set, construct a RANSAC-based random sampling sequence in the PnP algorithm solution framework, and calculate the initial solution of the pose parameters by minimizing the reprojection error function;

[0018] Input the initial solution of the pose parameters into the Levenberg-Marquardt optimizer, perform non-linear iterative solution on the rotation matrix and translation vector, and output the initial pose including six degrees of freedom.

[0019] In one embodiment, combine the generative adversarial network to perform illumination distortion elimination processing to generate the three-dimensional coordinates of the target object, including:

[0020] Construct a residual learning model based on the conditional generative adversarial network, and generate an illumination-invariant texture feature vector through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data;

[0021] Input the illumination-invariant texture feature vector and the initial pose parameters into the conditional generator, and construct an adversarial loss function through the Markov discriminator to iteratively generate the object surface reflectance map without illumination distortion;

[0022] Based on the pixel consistency constraint of the object surface reflectance map, perform multi-view triangulation reconstruction on the 3D point cloud data to generate the 3D coordinates of the target object including geometric topology.

[0023] In one embodiment, the initial solution of the pose parameters is calculated by minimizing the reprojection error function, and the following energy function model is used:

[0024]

[0025] where \(R\in SO(3)\) represents the rotation matrix, represents the translation vector, represents the camera perspective projection operator, and \(\pi([x,y,z] T ) = [f x x / z + c x , f y y / z + c y ) T , f x and \(f y \) represent the focal lengths, \(c x \) and \(c y \) represent the principal coordinate points, represents the camera intrinsic matrix, represents the coordinates of the \(i\)-th geometric feature point segmented from the 3D point cloud structured dataset, \(x i \in R 2 \) represents the 2D observation point coordinates of the RGB-D image matching with \(X i , represents the Geman-McClure robust kernel function.

[0026] In one embodiment, construct a multi-scale line feature detection model to perform iterative filtering on the 3D point cloud data, and use the following anisotropic diffusion equation:

[0027]

[0028] where \(I(x,y,t)\) represents the image intensity field smoothed by the Gaussian kernel \(\rho\), \(p\in[0.5,1.5]\), represents the diffusion coefficient function corrected based on the Weickert tensor, \(\sigma\) represents the edge preservation threshold, \(\alpha\in[1.5,2.0]\) represents the anisotropy index, represents the gradient field calculated after Gaussian smoothing of the image intensity field with a scale length of \(p\), and the iteration termination condition of this diffusion equation is k represents the number of iterations, and ||·|| F represents the Frobenius norm.

[0029] In one embodiment, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50 cm, 100 cm]. Each module synchronously acquires RGB-D images at a frame rate of 60 Hz. The data fusion process adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3 mm.

[0030] In a second aspect, the present application also provides a robotic arm positioning and grasping device based on machine vision, including:

[0031] A point cloud reconstruction module, configured to synchronously obtain RGB-D images of a target scene by using the multi-view camera array, and generate three-dimensional point cloud data through data fusion processing;

[0032] A pose optimization module, configured to use an improved LSD algorithm and a PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, and perform illumination distortion elimination processing in combination with a generative adversarial network to generate the three-dimensional coordinates of the target object;

[0033] An error transfer module, configured to construct a robotic arm motion error transfer model through Monte Carlo simulation based on the three-dimensional coordinates, and generate a set of candidate grasping schemes in combination with a reinforcement learning algorithm;

[0034] A grasping decision module, configured to comprehensively score and rank the set of candidate grasping schemes based on a preset priority evaluation rule, and select the scheme with the highest comprehensive score as the optimal grasping scheme;

[0035] A trajectory planning module, configured to generate a robotic arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme.

[0036] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned robotic arm positioning and grasping method based on machine vision are implemented.

[0037] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned robotic arm positioning and grasping method based on machine vision are implemented.

[0038] The above-mentioned robotic arm positioning and grasping method, device, computer equipment, and storage medium based on machine vision synchronously collect RGB-D image data through a multi-view camera array and fuse them to generate a three-dimensional point cloud. An improved LSD algorithm is used to construct a multi-scale line feature detection model, combined with the RANSAC optimization framework and Geman-McClure robust kernel function of the PnP algorithm, effectively suppressing feature matching deviations caused by local occlusion and texture loss, and reducing the initial pose estimation error. The three-dimensional point cloud is processed by a generative adversarial network to eliminate illumination distortion, and illumination-invariant texture features are generated based on a residual learning model to solve the problem of three-dimensional reconstruction distortion in complex illumination environments and improve the reconstruction accuracy of the object surface reflectivity. A motion error transfer model and a reinforcement learning decision-making mechanism are constructed in combination with Monte Carlo simulation. Through dynamic error compensation and priority evaluation rule screening, the robustness of the robotic arm grasping path planning scheme is improved under noise interference, achieving a high success rate of grasping in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a flowchart of a robotic arm positioning and grasping method based on machine vision according to the present invention;

[0041] Figure 2 It is a structural diagram of a robotic arm positioning and grasping device based on machine vision according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In order to make the purpose, technical solutions, and advantages of the present application clearer, the following further details the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0043] The implementation environment of the present invention includes a multi-view camera array, a six-degree-of-freedom industrial robotic arm, and an edge computing terminal or server. When there are complex operation requirements such as industrial sorting and flexible assembly, multi-modal data of the target object are obtained in real time through the camera array. After the edge computing terminal or server fuses them to generate a three-dimensional point cloud and calculates the optimal grasping pose, a joint motion trajectory instruction set is sent to the robotic arm to achieve the positioning and grasping of the target.

[0044] In one embodiment, as Figure 1As shown, a robotic arm positioning and grasping method based on machine vision is provided. In this embodiment, the method is exemplified by its application to an edge computing terminal. It can be understood that this method can also be applied to a server and can also be applied to a system including a computing terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0045] S01, Use a multi-view camera array to synchronously obtain RGB-D images of the target scene, and generate three-dimensional point cloud data through data fusion processing.

[0046] Among them, multiple binocular vision modules can be used, arranged in a circular pattern at a certain distance and angle to form a multi-view camera array, and synchronously obtain RGB-D (an image containing color (RGB) and depth (Depth) information) images of the target scene. Through data fusion processing such as establishing the mapping relationship between the RGB image and the depth image, using image registration to eliminate perspective deviation, and realizing data space alignment; segmenting the geometric structure of the target object and removing background interference; extracting the texture features of the object and generating texture feature vectors. Through spatio-temporal synchronous coding, the geometric boundary point coordinates, texture features, and timestamp information of the RGB-D image are fused to generate three-dimensional point cloud data, providing a high-precision three-dimensional model for subsequent pose calculation.

[0047] S02, Use the improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, and combine it with a generative adversarial network to perform illumination distortion elimination processing to generate the three-dimensional coordinates of the target object.

[0048] Among them, an improved LSD algorithm and a PnP algorithm are used for initial pose calculation, which can be achieved through: an anisotropic diffusion equation to construct a detection model, filtering the three-dimensional point cloud data to suppress noise while retaining the object contour edges, and generating a geometric feature point set of the target object; in the PnP solution framework, abnormal feature points are removed by random sampling, and based on minimizing the reprojection error, an initial solution of the pose parameters is calculated. For illumination distortion elimination based on a generative adversarial network, it can be achieved through: constructing a learning model, analyzing the normal vector distribution of the geometric feature point set of the three-dimensional point cloud to generate texture feature vectors independent of illumination changes, and stripping the influence of illumination on the surface reflection of the object; inputting the illumination-invariant texture features and the initial pose parameters into a conditional generator, and through constructing an adversarial loss function, iteratively generating a surface reflectance map of the object without illumination distortion to restore the true surface characteristics of the object; reconstructing the multi-viewpoint cloud data based on the reflectance map to generate the three-dimensional coordinates of the target object and achieve illumination-robust positioning. Through the multi-scale feature detection of the improved LSD algorithm and the robust optimization of the PnP algorithm, the pose estimation deviation caused by local occlusion and texture loss is suppressed; combined with the illumination distortion elimination of cGAN, the problem of three-dimensional reconstruction distortion under complex illumination is solved, and the positioning accuracy and environmental adaptability are improved.

[0049] S03. Based on the three-dimensional coordinates, a motion error transfer model of the robotic arm is constructed through Monte Carlo simulation, and a set of candidate grasping schemes is generated in combination with a reinforcement learning algorithm.

[0050] Among them, for the construction of the error transfer model based on Monte Carlo simulation, by analyzing multi-source interferences such as the joint motion error of the robotic arm and sensor noise, a probability distribution model is established for parameters such as the position error and angle deviation of each joint. Through the Monte Carlo simulation method, the kinematic model of the robotic arm is randomly sampled tens of thousands of times: each time the error parameters of each joint are sampled, the pose deviation of the end effector is calculated through forward kinematics, and the probability distribution of the end error is statistically analyzed to construct an error transfer model from the joint space to the task space to quantify the influence of the error on the grasping position. In the set of candidate grasping schemes generated by reinforcement learning, with the three-dimensional coordinates of the target object, the parameters of the error transfer model, and the current pose of the robotic arm as state variables, a multi-dimensional state space including geometric features (such as the normal vector of the grasping point and surface curvature) and error robustness indicators is constructed. The action is defined as the grasping scheme parameters, including the coordinates of the grasping point, the gripper attitude, the pre-grasping height, etc., and the action representation is achieved through discretization or continuous space parameterization. Combining grasping stability (such as contact area and frictional moment), error tolerance (the influence of end error on the grasping success rate), and motion feasibility (obstacle avoidance constraints), a multi-objective reward function is designed to guide the reinforcement learning agent to explore high-robustness grasping strategies. Reinforcement learning algorithms such as Deep Q-Network (DQN) or Proximal Policy Optimization (PPO) are used, with the error transfer model as the environment simulator, and through interacting with the virtual environment for learning, a set of candidate schemes including different grasping points and postures is generated.

[0051] S04, comprehensively score and rank the set of candidate grasping solutions based on a preset priority evaluation rule, and select the solution with the highest comprehensive score as the optimal grasping solution.

[0052] Among them, the preset priority evaluation rule may include the following evaluation indicators: grasping stability: based on geometric parameters such as the consistency of the normal vector of the grasping point, the contact area between the gripper and the object surface, and the frictional torque, evaluate the anti-disturbance ability during grasping; error tolerance: combine the error transfer model of Monte Carlo simulation to quantify the influence degree of the pose deviation of the end effector on the grasping success rate; motion feasibility: detect whether there is a collision risk in the motion trajectory of the robotic arm from the current pose to the grasping pose, and evaluate whether the joint angles exceed the physical limits; operation efficiency: calculate indicators such as the grasping path length and the joint motion energy consumption to optimize the operation efficiency in industrial scenarios; environmental adaptability: evaluate the robustness of the solution in the actual working conditions for interference factors such as light and vibration. Use the analytic hierarchy process (AHP) or the dynamic weight adjustment strategy to preset the priority weight coefficients of each indicator according to different operation scenarios (such as precision assembly and heavy object handling), calculate the comprehensive scores of the candidate solutions using a linear weighted model, sort the comprehensive scores of all candidate solutions in descending order, select the solution with the highest score as the optimal grasping solution, and at the same time output the score details of each solution for system debugging reference.

[0053] S05, generate the joint motion trajectory of the robotic arm and the corresponding control instruction set according to the parameter characteristics of the optimal grasping solution.

[0054] Among them, based on the joint space mapping of inverse kinematics, with the end pose (three-dimensional coordinates + attitude matrix) of the optimal grasping scheme as the input, an inverse kinematics model of the robotic arm can be established. An improved Levenberg-Marquardt (an iterative optimization algorithm for solving non-linear least squares problems) iterative algorithm is used, combined with the physical constraints of the robotic arm joints (angle range, speed limit), to solve the joint angle combination that satisfies the end pose. For the multi-solution problem of redundant degree-of-freedom robotic arms, the optimal joint solution is selected based on the following principles: the joint angles are close to the middle position to avoid reaching the physical limits; the joint movement range is minimized to reduce energy consumption; combined with the error transfer model, the joint combination with the lowest sensitivity to the end error is selected. The grasping process can be divided into three stages: Approach - Grasp - Withdraw. Five-degree polynomial interpolation is performed on the joint angles, speeds, and accelerations of each stage, and solved through endpoint constraints (position, speed, acceleration) to generate the joint motion trajectory of the robotic arm. At the same time, the dynamic model of the robotic arm can be introduced to perform dynamic smoothing on the trajectory: restricting the joint jerk to reduce motion shocks; based on the trajectory error feedback of the end effector, the polynomial parameters are adjusted in real time to compensate for the modeling errors. The trajectory is discretized into joint angle commands in a time series, and an instruction buffer queue is constructed. Real-time control is achieved through the following strategies: based on the deviation between the current pose of the robotic arm and the target trajectory, a proportional-integral-differential (PID) controller is used to dynamically adjust the joint output; integrating force feedback signals (such as the data of the grasping force sensor), when an abnormal contact is detected, trajectory replanning is triggered to avoid collisions or grasping failures.

[0055] The above method for robotic arm positioning and grasping based on machine vision synchronously acquires RGB-D images of the target scene using a multi-view camera array and fuses them to generate three-dimensional point cloud data. By combining the improved LSD algorithm and the PnP algorithm, it suppresses the feature matching deviation caused by local occlusion and texture loss, reduces the initial pose estimation error. At the same time, it uses a generative adversarial network to eliminate the illumination distortion to improve the reconstruction accuracy of the object surface reflectivity. Based on Monte Carlo simulation, it constructs a robotic arm motion error transfer model and combines reinforcement learning to generate candidate grasping schemes. Through a preset priority evaluation rule, the optimal scheme is selected to generate the joint motion trajectory and control instruction set of the robotic arm, effectively solving the problems of large target positioning deviation and sensitivity to illumination interference in complex environments, improving the grasping accuracy and robustness of the robotic arm, and being able to meet the requirements of high anti-interference positioning and grasping in complex scenarios such as industrial sorting and flexible assembly.

[0056] In one of the embodiments, the RGB-D image of the target scene generates three-dimensional point cloud data through data fusion processing, including:

[0057] S11. Establish the mapping relationship of the RGB-D image of the target scene, align the RGB-D image using an image registration algorithm, and obtain the aligned image;

[0058] S12. Use a geometric segmentation method based on point cloud normal vector and curvature analysis to segment the target object in the aligned image, and obtain the segmented image;

[0059] S13. Use local binary pattern to extract texture features of the target object in the segmented image, and obtain the texture feature vector;

[0060] S14. Perform spatio-temporal synchronous coding on the geometric boundary point coordinates of the segmented image and the corresponding texture feature vector, and fuse them to generate a three-dimensional point cloud structured dataset containing geometric, texture, and timestamp information.

[0061] Specifically, the specific process of generating three-dimensional point cloud data from the RGB-D image of the target scene through data fusion processing is as follows: establish the mapping relationship between the RGB image and the depth image, use an image registration algorithm to eliminate the perspective deviation to achieve the spatial alignment of multi-modal data, and obtain the aligned image; use a geometric segmentation method based on point cloud normal vector and curvature analysis, calculate the local normal vector distribution and curvature change characteristics of the point cloud, and segment the target object from the background to obtain the segmented image; use local binary pattern (LBP) to extract texture features of the target object in the segmented image, and generate a feature vector representing the surface texture information; perform spatio-temporal synchronous coding on the three-dimensional coordinates of the geometric boundary points of the segmented image and the corresponding texture feature vector, and fuse them to generate a three-dimensional point cloud structured dataset containing the object's geometric structure, surface texture, and timestamp information, providing multi-dimensional data support with both spatial accuracy and texture features for subsequent pose calculation.

[0062] In one embodiment, use the improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, including:

[0063] S21. Based on the Gaussian kernel function to correct the anisotropic diffusion equation, construct a multi-scale line feature detection model, perform iterative filtering on the three-dimensional point cloud data, and generate a geometric feature point set of the target object contour;

[0064] S22. According to the geometric feature point set, construct a random sampling sequence based on RANSAC in the PnP algorithm solution framework, and calculate the initial solution of the pose parameters by minimizing the reprojection error function;

[0065] S23. Input the initial solution of the pose parameters into the Levenberg-Marquardt optimizer, perform non-linear iterative solution on the rotation matrix and translation vector, and output the initial pose containing six degrees of freedom.

[0066] Specifically, in this embodiment, the process of calculating the initial pose of the target object using the improved LSD algorithm and PnP algorithm is as follows: Based on the Gaussian kernel function, the anisotropic diffusion equation is corrected to construct a multi-scale line feature detection model. Through iterative filtering of the three-dimensional point cloud data, while suppressing noise, the geometric features of the target object's contour are retained, and an accurate contour feature point set is generated. Based on this feature point set, a random sampling sequence based on RANSAC (Random Sample Consensus) is constructed within the PnP algorithm solution framework. By minimizing the reprojection error function containing Geman-McClure (a robust kernel function), the pose parameters, that is, the initial solutions of the rotation matrix and translation vector, are calculated, and the interference of abnormal feature points is eliminated. The initial solutions are input into the Levenberg-Marquardt optimizer to perform non-linear iterative optimization on the rotation matrix and translation vector, and the six-degree-of-freedom initial pose including three-dimensional translation and three-dimensional rotation is output, realizing high-precision estimation of the target object's pose.

[0067] In one of the embodiments, in combination with the generative adversarial network, light distortion elimination processing is performed to generate the three-dimensional coordinates of the target object, including:

[0068] S31, construct a residual learning model based on the conditional generative adversarial network, and generate a light-invariant texture feature vector through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data;

[0069] S32, input the light-invariant texture feature vector and the initial pose parameters into the conditional generator, and construct an adversarial loss function through the Markov discriminator to iteratively generate the object surface reflectivity map without light distortion;

[0070] S33, based on the pixel consistency constraint of the object surface reflectivity map, perform multi-view triangulation reconstruction on the three-dimensional point cloud data to generate the three-dimensional coordinates of the target object including the geometric topology structure.

[0071] Exemplarily, the process of eliminating illumination distortion and generating three-dimensional coordinates in combination with a generative adversarial network is as follows: construct a residual learning model based on a conditional generative adversarial network (cGAN), analyze the normal vector distribution of the geometric feature point set of the three-dimensional point cloud, strip the influence of illumination changes on surface reflection, and generate a texture feature vector independent of illumination; input the illumination-invariant texture feature vector and the initial pose parameters into the conditional generator, construct an adversarial loss function using a Markov discriminator, and through the iterative game between the generator and the discriminator, generate a surface reflectivity map of the object with illumination distortion removed, restoring the true reflection characteristics of the object; based on the pixel consistency constraint of the reflectivity map, perform triangulation reconstruction on the multi-viewpoint cloud data to generate the three-dimensional coordinates of the target object including the geometric topology structure, achieving high-precision three-dimensional positioning in a complex illumination environment.

[0072] In one embodiment, S41, calculate the initial solution of the pose parameters by minimizing the reprojection error function, using the following energy function model:

[0073]

[0074] where, R ∈ SO(3) represents the rotation matrix, represents the translation vector, represents the camera perspective projection operator, and π([x, y, z] T ) = [f x x / z + c x , f y y / z + c y ) T , f x and f y represent the focal lengths, c x and c y represent the principal coordinate points, represents the camera intrinsic matrix, represents the coordinate of the i-th geometric feature point segmented from the three-dimensional point cloud structured dataset, x i ∈ R 2 represents the two-dimensional observation point coordinate that the RGB-D image matches with X i , represents the Geman-McClure robust kernel function.

[0075] Specifically, this formula constructs an optimization objective function for the pose parameters (rotation matrix (R), translation vector (t)) by quantifying the projection error of the three-dimensional point cloud feature points on the camera imaging plane. The rotation matrix R ∈ SO(3) characterizes the rotation pose of the target object in three-dimensional space, and the translation vector characterizes the position offset of the target object in the camera coordinate system. [R|t] constitutes the camera extrinsic matrix, realizing the transformation from the world coordinate system to the camera coordinate system, and the camera intrinsic matrix including the focal length f x and f y and the principal coordinate points (c x , c y ) define the imaging projection relationship. The projection operator π projects the three-dimensional space point X i onto the two-dimensional image plane to obtain the theoretical projection point π(K[R|t]X i ). The squared Euclidean distance between the actually observed two-dimensional observation point coordinates x i and the theoretical projection point is the reprojection error of a single feature point. The Geman-McClure kernel function is defined as When the error s is small, it is approximately linearly weighted to retain the error contribution of valid feature points; when the error s is large, ρ(s) → 1, the weight saturates, and the influence of outliers (such as mismatched features and noise points) is suppressed. In the PnP algorithm solution framework, this kernel function is used in combination with RANSAC (Random Sample Consensus): RANSAC generates candidate pose solutions by randomly sampling three-dimensional to two-dimensional point pairs; uses ρ(s) to weight the error of each feature point and calculates the global energy function E(R, t); iteratively filters the pose solution that minimizes E(R, t), and at the same time eliminates the feature points determined to be outliers by ρ(s) to achieve the joint optimization of pose estimation and outlier filtering.

[0076] In one embodiment, S51, construct a multi-scale line feature detection model to perform iterative filtering on the three-dimensional point cloud data using the following anisotropic diffusion equation:

[0077]

[0078] where I(x, y, t) represents the image intensity field smoothed by the Gaussian kernel ρ, p ∈ [0.5, 1.5], represents the diffusion coefficient function corrected based on the Weickert tensor, σ represents the edge retention threshold, α ∈ [1.5, 2.0] represents the anisotropy index, represents the gradient field calculated after Gaussian smoothing of the image intensity field with a scale length of p, and the iteration termination condition of this diffusion equation is k represents the number of iterations, ||·|| F represents the Frobenius norm.

[0079] Exemplarily, this equation realizes multi-scale filtering of the three-dimensional point cloud through a gradient-driven diffusion process. For the diffusion direction, it diffuses along the direction perpendicular to the gradient of the image intensity field I to retain the edge features; for the diffusion intensity adjustment, it is adjusted through the diffusion coefficient function g σAdjust the diffusion amount adaptively to achieve a balance between edge protection and noise suppression. Its adaptive diffusion process is as follows: For the smoothing stage: When (such as in the noise area), g σ (s) ≈ 1, the diffusion intensity is large, effectively suppressing random noise; For the edge retention stage: When (such as the object contour), g σ (s) ≈ 0, the diffusion is suppressed, maintaining edge sharpness. By adjusting p ∈ [0.5, 1.5] and iteratively calculating within the range, it is achieved that: at a fine scale (p → 0.5), capturing fine structures and sharp edges; at a coarse scale (p → 1.5), extracting the overall contour and main structures; by superimposing gradient fields at different scales, a point cloud representation containing multi-resolution line features is generated. The iteration termination condition of this formula is to ensure that the diffusion process terminates when the change in the intensity field is less than 10 -3 . By fusing the mathematical anisotropic diffusion theory and the physical geometric features of the point cloud, the problem of feature mis-matching in traditional methods in scenarios with texture loss and illumination changes is solved.

[0080] In one embodiment, S61, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50 cm, 100 cm]. Each module synchronously acquires RGB-D images at a frame rate of 60 Hz. The data fusion process adopts a spatial down-sampling strategy based on voxel hashing, and the point cloud resolution is set to 3 mm.

[0081] Specifically, 8 binocular vision modules are arranged in a ring, with a radius range of 50 cm to 100 cm. Its technical advantages are as follows: The 8 modules form a 360° circular field of view, which can synchronously collect RGB-D data of the target object from all directions, solving the occlusion blind area problem of traditional single / binocular cameras, and is especially suitable for the all-round modeling of objects with complex postures; The adjustable radius design (50 cm to 100 cm) adapts to different sizes of targets (such as small parts to medium-sized workpieces), and by adjusting the array radius, the spatial resolution and the field of view can be balanced. The high frame rate of 60 Hz can effectively suppress image blur caused by the movement of the robotic arm or the jitter of the target object; Through the hardware trigger synchronization mechanism, the time stamp deviation of multi-view data can be ensured to be ≤ 1 ms, providing an accurate spatio-temporal reference for subsequent point cloud fusion and avoiding three-dimensional reconstruction distortion caused by time misalignment. Adopting a spatial downsampling strategy based on voxel hashing, the point cloud resolution is set to 3 mm. Through voxelization, the original point cloud (usually containing more than 1 million points) is compressed to the 100,000 level, and the computing efficiency is increased by 80%. At the same time, geometric features are retained. For example, for 3 mm voxels, the retention rate of edge points with a curvature change rate > 0.1 rad / mm reaches 92%; Resolution adaptation: The 3 mm resolution takes into account both the industrial grasping accuracy (the positioning error at the end of the robotic arm ≤ 0.5 mm) and the consumption of computing resources. At this resolution, the contour error of the object reconstructed from the point cloud ≤ 0.8 mm, which can meet the requirements of the precision assembly scenario.

[0082] The above-mentioned robotic arm positioning and grasping method based on machine vision synchronously acquires RGB-D images of the target scene by using a multi-view camera array and generates a three-dimensional point cloud structured dataset containing geometric, texture, and timestamp information through data fusion. It constructs a multi-scale line feature detection model by combining the anisotropic diffusion equation modified based on the Gaussian kernel function in the improved LSD algorithm to iteratively filter and generate a set of geometric feature points of the target object's contour. It calculates the initial solution of the pose parameters by minimizing the reprojection error with the random sampling sequence based on RANSAC and the Geman-McClure robust kernel function in the PnP algorithm framework, and outputs the initial six-degree-of-freedom pose in cooperation with the Levenberg-Marquardt optimizer. At the same time, it generates an illumination-invariant texture feature vector through the residual learning model of the conditional generative adversarial network and iteratively generates an object surface reflectivity map without illumination distortion. It reconstructs the three-dimensional coordinates of the target object based on pixel consistency constraints, constructs a robotic arm motion error transfer model based on Monte Carlo simulation, and generates a set of candidate grasping schemes in combination with reinforcement learning. It screens the optimal scheme through a preset priority evaluation rule and generates a joint motion trajectory and a control instruction set, effectively solving the technical problems of large target positioning deviation and sensitivity to illumination interference in complex environments. It suppresses the feature matching deviation caused by local occlusion and texture loss through multi-modal data fusion and non-linear optimization algorithms, eliminates the impact of illumination distortion on three-dimensional reconstruction, and improves the grasping accuracy and robustness of the robotic arm by combining error transfer modeling and reinforcement learning decision-making, realizing high-success-rate positioning and grasping in scenarios such as industrial sorting and flexible assembly.

[0083] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0084] Based on the same inventive concept, an embodiment of the present application further provides a robotic arm positioning and grasping device based on machine vision for implementing the above-mentioned robotic arm positioning and grasping method based on machine vision. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the robotic arm positioning and grasping device based on machine vision provided below can refer to the limitations on the robotic arm positioning and grasping method based on machine vision in the above text, and will not be elaborated here.

[0085] In an exemplary embodiment, as Figure 2 shown, a robotic arm positioning and grasping device based on machine vision is provided, including:

[0086] A point cloud reconstruction module 101, configured to synchronously acquire RGB-D images of a target scene by using a multi-view camera array, and generate three-dimensional point cloud data through data fusion processing;

[0087] A pose optimization module 102, configured to calculate the initial pose of a target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and perform illumination distortion elimination processing in combination with a generative adversarial network to generate the three-dimensional coordinates of the target object;

[0088] An error transfer module 103, configured to construct a robotic arm motion error transfer model through Monte Carlo simulation based on the three-dimensional coordinates, and generate a set of candidate grasping schemes in combination with a reinforcement learning algorithm;

[0089] A grasping decision module 104, configured to comprehensively score and rank the set of candidate grasping schemes based on a preset priority evaluation rule, and select the scheme with the highest comprehensive score as the optimal grasping scheme;

[0090] A trajectory planning module 105, configured to generate a robotic arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme.

[0091] In one embodiment, the point cloud reconstruction module 101 is further configured to:

[0092] Establish a mapping relationship of the RGB-D images of the target scene, and align the RGB-D images by using an image registration algorithm to obtain the aligned images;

[0093] Use a geometric segmentation method based on point cloud normal vector and curvature analysis to segment the target object in the aligned image to obtain the segmented image;

[0094] Use local binary pattern to perform texture feature extraction processing on the target object in the segmented image to obtain a texture feature vector;

[0095] Spatially and temporally synchronously encode the geometric boundary point coordinates of the segmented image with the corresponding texture feature vectors, and fuse them to generate a three-dimensional point cloud structured dataset containing geometric, texture, and timestamp information.

[0096] In one embodiment, the pose optimization module 102 is further configured to:

[0097] Based on the Gaussian kernel function, correct the anisotropic diffusion equation, construct a multi-scale line feature detection model, perform iterative filtering processing on the three-dimensional point cloud data, and generate a set of geometric feature points of the target object contour;

[0098] According to the set of geometric feature points, construct a random sampling sequence based on RANSAC in the PnP algorithm solution framework, and calculate the initial solution of the pose parameters by minimizing the reprojection error function;

[0099] Input the initial solution of the pose parameters into the Levenberg-Marquardt optimizer, perform nonlinear iterative solution on the rotation matrix and the translation vector, and output the initial pose including six degrees of freedom.

[0100] In one embodiment, the pose optimization module 102 is further configured to:

[0101] Construct a residual learning model based on the conditional generative adversarial network, and generate illumination-invariant texture feature vectors through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data;

[0102] Input the illumination-invariant texture feature vectors and the initial pose parameters into the conditional generator, construct an adversarial loss function through the Markov discriminator, and iteratively generate an object surface reflectance map with removed illumination distortion;

[0103] Based on the pixel consistency constraint of the object surface reflectance map, perform multi-view triangulation reconstruction on the three-dimensional point cloud data, and generate the three-dimensional coordinates of the target object including the geometric topology structure.

[0104] In one embodiment, the pose optimization module 102 is further configured to calculate the initial solution of the pose parameters using the following energy function model:

[0105]

[0106] where \(R\in SO(3)\) represents the rotation matrix, represents the translation vector, represents the camera perspective projection operator, and \(\pi([x,y,z] T ) = [f x x / z + c x , f y y / z + c y T , f x ​and f y represents the focal length, c x and c y represent the principal coordinate points, represents the camera intrinsic matrix, represents the coordinates of the i-th geometric feature point segmented from the 3D point cloud structured dataset, x i ∈R 2 represents the 2D observation point coordinates where the RGB-D image matches X i represents the Geman-McClure robust kernel function.

[0107] In one embodiment, the pose optimization module 102 is further configured to perform iterative filtering processing on the 3D point cloud data using the following anisotropic diffusion equation:

[0108]

[0109] where I(x, y, t) represents the image intensity field smoothed by the Gaussian kernel ρ, p ∈ [0.5, 1.5], represents the diffusion coefficient function corrected based on the Weickert tensor, σ represents the edge-preserving threshold, α ∈ [1.5, 2.0] represents the anisotropy index, represents the gradient field calculated after Gaussian smoothing of the image intensity field with a scale length of p, and the iteration termination condition of this diffusion equation is k represents the number of iterations, ||·|| F represents the Frobenius norm.

[0110] In one embodiment, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50 cm, 100 cm]. Each module synchronously acquires RGB-D images at a frame rate of 60 Hz, and the data fusion processing adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3 mm.

[0111] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of a robotic arm positioning and grasping method based on machine vision as described above are implemented.

[0112] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0113] ​For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0114] The above embodiments only represent several implementation manners of the embodiments of the present application. The descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A robotic arm positioning and grasping method based on machine vision, characterized in that, The method includes: Synchronously acquiring RGB-D images of a target scene using a multi-view camera array, and generating three-dimensional point cloud data through data fusion processing; Using an improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, and performing illumination distortion elimination processing in combination with a generative adversarial network to generate the three-dimensional coordinates of the target object; Based on the three-dimensional coordinates, constructing a manipulator motion error transfer model through Monte Carlo simulation, and generating a set of candidate grasping schemes in combination with a reinforcement learning algorithm; Comprehensively scoring and sorting the set of candidate grasping schemes based on a preset priority evaluation rule, and selecting the scheme with the highest comprehensive score as the optimal grasping scheme; Generating a manipulator joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme.

2. The method according to claim 1, wherein The RGB-D images of the target scene generate three-dimensional point cloud data through data fusion processing, including: Establishing a mapping relationship of the RGB-D images of the target scene, and aligning the RGB-D images using an image registration algorithm to obtain the aligned images; Using a geometric segmentation method based on point cloud normal vector and curvature analysis to segment the target object in the aligned images to obtain the segmented images; Using local binary pattern to perform texture feature extraction processing on the target object in the segmented images to obtain texture feature vectors; Performing spatio-temporal synchronous coding on the geometric boundary point coordinates of the segmented images and the corresponding texture feature vectors, and fusing them to generate a three-dimensional point cloud structured data set containing geometric, texture, and timestamp information.

3. The method according to claim 2, wherein The use of an improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data includes: Based on the Gaussian kernel function to correct the anisotropic diffusion equation, constructing a multi-scale line feature detection model, and performing iterative filtering processing on the three-dimensional point cloud data to generate a geometric feature point set of the target object contour; According to the geometric feature point set, constructing a random sampling sequence based on RANSAC in the PnP algorithm solution framework, and calculating the initial solution of the pose parameters by minimizing the reprojection error function; Inputting the initial solution of the pose parameters into a Levenberg-Marquardt optimizer, and performing non-linear iterative solution on the rotation matrix and translation vector to output the initial pose including six degrees of freedom.

4. The method according to claim 1, wherein The combination of a generative adversarial network for illumination distortion elimination processing to generate the three-dimensional coordinates of the target object includes: Constructing a residual learning model based on a conditional generative adversarial network, and generating an illumination-invariant texture feature vector through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data; Inputting the illumination-invariant texture feature vector and the initial pose parameters into a conditional generator, and constructing an adversarial loss function through a Markov discriminator to iteratively generate an object surface reflectance map without illumination distortion; Based on the pixel consistency constraint of the object surface reflectance map, performing multi-view triangulation reconstruction on the three-dimensional point cloud data to generate the three-dimensional coordinates of the target object including geometric topology.

5. The method according to claim 3, wherein The initial solution of the pose parameters is calculated by minimizing the reprojection error function, and the following energy function model is used: where \(R\in SO(3)\) represents the rotation matrix, represents the translation vector, represents the camera perspective projection operator, and \(\pi([x,y,z] T ) = [f x x / z + c x , f y y / z + c y ) T , f x and \(f y represent the focal lengths, \(c x and \(c y represent the principal coordinate points, represents the camera intrinsic matrix, represents the coordinates of the \(i\)-th geometric feature point segmented from the 3D point cloud structured dataset, \(x i \in\mathbb{R} 2 represents the coordinates of the 2D observation point in the RGB-D image that matches \(X i , represents the Geman - McClure robust kernel function.

6. The method according to claim 3, wherein The multi-scale line feature detection model is constructed to perform iterative filtering on the three-dimensional point cloud data, and the following anisotropic diffusion equation is used: Among them, I(x, y, t) represents the image intensity field after being smoothed by the Gaussian kernel ρ, p ∈ [0.5, 1.5], represents the diffusion coefficient function corrected based on the Weickert tensor, σ represents the edge-preserving threshold, α ∈ [1.5, 2.0] represents the anisotropy index, represents the gradient field calculated after performing Gaussian smoothing with a scale length of p on the image intensity field. The iteration termination condition for this diffusion equation is k represents the number of iterations, ||·|| F represents the Frobenius norm.

7. The method according to claim 1, wherein The multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50 cm, 100 cm]. Each module synchronously acquires RGB-D images at a frame rate of 60 Hz. The data fusion process adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3 mm.

8. A robotic arm positioning and grasping device based on machine vision, characterized in that, The device includes: A point cloud reconstruction module, configured to synchronously obtain RGB-D images of a target scene by using the multi-view camera array, and generate three-dimensional point cloud data through data fusion processing; A pose optimization module, configured to calculate the initial pose of a target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and perform illumination distortion elimination processing in combination with a generative adversarial network to generate the three-dimensional coordinates of the target object; An error transfer module, configured to construct a manipulator motion error transfer model through Monte Carlo simulation based on the three-dimensional coordinates, and generate a set of candidate grasping schemes in combination with a reinforcement learning algorithm; A grasping decision module, configured to comprehensively score and sort the set of candidate grasping schemes based on a preset priority evaluation rule, and select the scheme with the highest comprehensive score as the optimal grasping scheme; A trajectory planning module, configured to generate a manipulator joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images