A machine vision-based mechanical arm positioning and grabbing method

Abstract

The application discloses a kind of mechanical arm positioning and grabbing method based on machine vision, it is related to machine vision and mechanical arm control technical field, the method is by multi-view camera array synchronization acquisition target scene's RGB-D image, three-dimensional point cloud data is generated after data fusion;Improved LSD algorithm and PnP algorithm are used to calculate the initial pose of target object, combined with generated adversarial network to eliminate light distortion, output three-dimensional coordinates;Based on Monte Carlo simulation, a mechanical arm motion error transfer model is constructed, and a candidate set of grabbing schemes is generated using reinforcement learning;The optimal grabbing scheme is selected by a pre-set priority evaluation rule, and a mechanical arm joint motion trajectory and control instruction set are generated.The application solves the technical problems of large target positioning deviation and light interference sensitivity in complex environments through multi-modal data fusion and nonlinear optimization algorithm, and improves the precision and robustness of mechanical arm grabbing.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision and mechanical arm control, and particularly relates to a mechanical arm positioning and grabbing method based on machine vision BACKGROUND

[0002] With the rapid development of machine vision and industrial automation technology, the mechanical arm grabbing technology based on vision guidance gradually becomes a key research direction in the field of intelligent manufacturing. The traditional method usually acquires two-dimensional image information of the target object by using a monocular or binocular camera, realizes object positioning through feature matching and geometric reasoning, and generates a mechanical arm motion trajectory based on a preset grabbing strategy. For example, the RGB-D camera is generally used in the prior art to acquire depth information, the pose parameters are iteratively optimized through point cloud registration and ICP algorithm, and the grabbing path is planned in combination with the inverse kinematics solution. However, the current traditional method has significant limitations: in a complex lighting environment, the reflection characteristics of the object surface easily lead to depth data noise and RGB image color distortion, and the existing data fusion method is difficult to effectively suppress the influence of lighting distortion on the three-dimensional reconstruction accuracy; the pose estimation algorithm based on a single geometric feature (such as SIFT and ORB) is prone to feature matching errors in the scene of local occlusion or texture missing of the object, resulting in the accumulation and amplification of pose estimation deviation; the motion error transmission modeling of the mechanical arm usually relies on linear approximation or static parameter assumption, and is difficult to dynamically adapt to multi-source interference in actual work, resulting in insufficient robustness of the grabbing scheme. These problems seriously restrict the application effect of the mechanical arm in the scenes of industrial sorting and flexible assembly, and it is urgent to develop an integrated positioning and grabbing solution with high anti-interference capability. SUMMARY

[0003] Therefore, it is necessary to provide a mechanical arm positioning and grabbing method based on machine vision which can solve the above problems.

[0004] In a first aspect, the application provides a mechanical arm positioning and grabbing method based on machine vision, comprising:

[0005] synchronously acquiring RGB-D images of a target scene by using a multi-view camera array, and generating three-dimensional point cloud data through data fusion processing;

[0006] calculating the initial pose of the target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and eliminating lighting distortion by using a generative adversarial network to generate three-dimensional coordinates of the target object;

[0007] based on the three-dimensional coordinates, constructing a mechanical arm motion error transmission model through Monte Carlo simulation, and generating a candidate grabbing scheme set in combination with a reinforcement learning algorithm;

[0008] The candidate crawling scheme set is comprehensively scored and ranked based on preset priority evaluation rules, and a scheme with the highest comprehensive score is selected as an optimal crawling scheme.

[0009] According to the parameter characteristics of the optimal crawling scheme, a joint motion trajectory of the mechanical arm and a corresponding control instruction set are generated.

[0010] In one of the embodiments, the RGB-D image of the target scene generates three-dimensional point cloud data through data fusion processing, including:

[0011] The mapping relationship of the RGB-D image of the target scene is established, an image registration algorithm is used to align the RGB-D image, and an aligned image is obtained.

[0012] A geometric segmentation method based on point cloud normal vector and curvature analysis is used to segment the target object in the aligned image, and a segmented image is obtained.

[0013] A local binary pattern is used to extract texture features of the target object in the segmented image, and a texture feature vector is obtained.

[0014] The geometric boundary point coordinates of the segmented image and the corresponding texture feature vector are spatiotemporally synchronized and encoded, and a three-dimensional point cloud structured data set containing geometric, texture and timestamp information is fused and generated.

[0015] In one of the embodiments, an improved LSD algorithm and PnP algorithm are used to calculate the initial pose of the target object in the three-dimensional point cloud data, including:

[0016] Based on the anisotropic diffusion equation corrected by the Gaussian kernel function, a multi-scale line feature detection model is constructed, and the three-dimensional point cloud data is iteratively filtered to generate a geometric feature point set of the target object contour.

[0017] According to the geometric feature point set, a random sampling sequence based on RANSAC is constructed in the PnP algorithm solving framework, and the initial solution of the pose parameters is calculated by minimizing the re-projection error function.

[0018] The initial solution of the pose parameters is input into the Levenberg-Marquardt optimizer, and the rotation matrix and translation vector are nonlinearly iteratively solved to output an initial pose containing six degrees of freedom.

[0019] In one of the embodiments, a generative adversarial network is combined to perform illumination distortion elimination processing to generate three-dimensional coordinates of the target object, including:

[0020] A residual learning model based on conditional generative adversarial network is constructed, and a light-invariant texture feature vector is generated through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data.

[0021] The light invariance texture feature vector is input into an initial pose parameter input condition generator, an adversarial loss function is constructed through a Markov discriminator, and an object surface reflectance map without light distortion is iteratively generated;

[0022] Based on the pixel consistency constraint of the object surface reflectance map, multi-view triangulation reconstruction is performed on the three-dimensional point cloud data to generate a three-dimensional coordinate of the target object containing geometric topological structure.

[0023] In one of the embodiments, an initial solution of the pose parameter is calculated by minimizing a re-projection error function, using the following energy function model:

[0024]

[0025] wherein R∈SO(3) represents a rotation matrix, represents a translation vector, represents a camera perspective projection operator, and π([x, y, z] T ) = [f x x / z + c x , f y y / z + c y ] T , f x and f y represent focal lengths, c x and c y represent principal points, represents a camera intrinsic matrix, represents the i-th geometric feature point coordinate segmented from the three-dimensional point cloud structured data, x i ∈R 2 represents a two-dimensional observation point coordinate matched with X i , represents a Geman-McClure robust kernel function.

[0026] In one of the embodiments, a multi-scale line feature detection model is constructed to perform iterative filtering processing on the three-dimensional point cloud data, using the following anisotropic diffusion equation:

[0027]

[0028] wherein I(x, y, t) represents an image intensity field smoothed by a Gaussian kernel p, p∈[0.5, 1.5], represents a diffusion coefficient function based on Weickert tensor correction, σ represents an edge preservation threshold, and a∈[1.5, 2.0] represents an anisotropy index, represents a gradient field calculated after Gaussian smoothing of the image intensity field with a scale length of p, and the iteration termination condition of the diffusion equation is k represents the number of iterations, ||·|| F represents the Frobenius norm. F k represents the number of iterations, ||·|| F represents the Frobenius norm.

[0029] In one of the embodiments, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50cm, 100cm], each module synchronously collects RGB-D images at a frame rate of 60Hz, and the data fusion processing adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3mm.

[0030] In a second aspect, the application further provides a mechanical arm positioning and grabbing device based on machine vision, comprising:

[0031] A point cloud reconstruction module is configured to synchronously acquire RGB-D images of a target scene by using a multi-view camera array, and generate three-dimensional point cloud data through data fusion processing;

[0032] A pose optimization module is configured to calculate an initial pose of a target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and perform light distortion elimination processing in combination with a generative adversarial network to generate three-dimensional coordinates of the target object;

[0033] An error transmission module is configured to construct a mechanical arm motion error transmission model by Monte Carlo simulation based on the three-dimensional coordinates, and generate a candidate grabbing scheme set in combination with a reinforcement learning algorithm;

[0034] A grabbing decision module is configured to comprehensively score and sort the candidate grabbing scheme set based on a preset priority evaluation rule, and select a scheme with the highest comprehensive score as an optimal grabbing scheme;

[0035] A trajectory planning module is configured to generate a mechanical arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grabbing scheme.

[0036] In a third aspect, the application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned mechanical arm positioning and grabbing method based on machine vision when executing the computer program.

[0037] In a fourth aspect, the application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the above-mentioned mechanical arm positioning and grabbing method based on machine vision.

[0038] The machine vision-based mechanical arm positioning and grabbing method, device, computer equipment and storage medium, through the multi-view camera array synchronously collecting RGB-D image data and fusing to generate three-dimensional point cloud, using the improved LSD algorithm to construct a multi-scale line feature detection model, combining the RANSAC optimization framework of PnP algorithm and the Geman-McClure robust kernel function, the feature matching deviation caused by local occlusion and texture missing is inhibited, and the initial pose estimation error is reduced; the three-dimensional point cloud is processed by the generative adversarial network to eliminate illumination distortion, and the illumination invariant texture feature is generated based on the residual learning model, solving the problem of three-dimensional reconstruction distortion in complex lighting environment, and improving the object surface reflectivity reconstruction accuracy; combining the Monte Carlo simulation to construct the motion error transmission model and the reinforcement learning decision mechanism, through dynamic error compensation and priority evaluation rule screening, the robustness of the mechanical arm grabbing path planning scheme under noise interference is improved, and high success rate grabbing in complex scenes is realized. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiment or related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 A flow chart of a machine vision-based mechanical arm positioning and grabbing method of the present application;

[0041] Figure 2 A structural diagram of a machine vision-based mechanical arm positioning and grabbing device of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0043] The implementation environment of the present application includes a multi-view camera array, a six-degree-of-freedom industrial robot and an edge computing terminal or server. When there is a complex operation requirement such as industrial sorting and flexible assembly, the multi-modal data of the target object is obtained in real time by the camera array, the three-dimensional point cloud is generated by the edge computing terminal or server, and the optimal grabbing pose is calculated, then the joint motion trajectory instruction set is sent to the robot, and the positioning and grabbing of the target are realized.

[0044] In one embodiment, as Figure 1As shown, a machine vision-based robotic arm positioning and grabbing method is provided, and in this embodiment, the method is applied to an edge computing terminal for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a computing terminal and a server, and is implemented through the interaction of the terminal and the server. In this embodiment, the method includes the following steps:

[0045] S01, synchronously acquiring an RGB-D image of a target scene by using a multi-view camera array, and generating three-dimensional point cloud data through data fusion processing.

[0046] Among them, a plurality of binocular vision modules can be used, which are arranged in a ring according to a certain distance and angle, to form a multi-view camera array, to synchronously acquire an RGB-D (an image containing color (RGB) and depth (Depth) information) image of a target scene, to eliminate perspective deviation by using image registration through data fusion processing such as establishing a mapping relationship between an RGB image and a depth image, to realize data space alignment; to segment the geometric structure of the target object and remove background interference; to extract the texture features of the object and generate a texture feature vector, and to generate three-dimensional point cloud data by fusing the geometric boundary point coordinates, texture features and timestamp information of the RGB-D image through space-time synchronous coding, to provide a high-precision three-dimensional model for subsequent pose calculation.

[0047] S02, using an improved LSD algorithm and PnP algorithm to calculate the initial pose of the target object in the three-dimensional point cloud data, and combining a generative adversarial network to eliminate light distortion, to generate three-dimensional coordinates of the target object.

[0048] Wherein, the improved LSD algorithm and PnP algorithm are used for initial pose calculation, the anisotropic diffusion equation is used to construct a detection model, the three-dimensional point cloud data is filtered to suppress noise and retain the object contour edge, and the geometric feature point set of the target object is generated; in the PnP solving framework, abnormal feature points are removed by random sampling, the re-projection error is minimized, and the initial solution of the pose parameters is calculated. The illumination distortion elimination based on the generative adversarial network can be achieved by: constructing a learning model, analyzing the normal vector distribution of the three-dimensional point cloud geometric feature point set, generating texture feature vectors independent of illumination changes, and stripping the influence of illumination on object surface reflection; the illumination invariant texture features and initial pose parameters are input into the conditional generator, an adversarial loss function is constructed, and the object surface reflectance map without illumination distortion is iteratively generated to restore the true surface characteristics of the object; based on the reflectance map, multi-view point cloud data is reconstructed to generate the three-dimensional coordinates of the target object, and the illumination robust positioning is realized. Through the multi-scale feature detection of the improved LSD algorithm and the robust optimization of the PnP algorithm, the pose estimation deviation caused by local occlusion and texture missing is suppressed; combined with the illumination distortion elimination of the cGAN, the three-dimensional reconstruction distortion problem under complex illumination is solved, and the positioning accuracy and environmental adaptability are improved.

[0049] S03, based on three-dimensional coordinates, a mechanical arm motion error transmission model is constructed by Monte Carlo simulation, and a candidate grasping scheme set is generated by combining a reinforcement learning algorithm.

[0050] Wherein, the error transmission model construction based on Monte Carlo simulation can analyze the joint motion error of the mechanical arm, sensor noise and other multi-source interference, establish a probability distribution model for each joint position error and angle deviation parameter, and through Monte Carlo simulation method, randomly sample the kinematics model of the mechanical arm thousands of times: each time the joint error parameter is sampled, the pose deviation of the end effector is calculated through forward kinematics, the probability distribution of the end error is counted, and the error transmission model from the joint space to the task space is constructed to quantify the influence of the error on the grasping position. In the generation of the candidate grasping scheme set by the reinforcement learning, the three-dimensional coordinates of the target object, the error transmission model parameters and the current pose of the mechanical arm are used as state variables, a multi-dimensional state space including geometric features (such as grasping point normal vector, surface curvature) and error robustness index is constructed. The action is defined as the grasping scheme parameter, including the grasping point coordinates, the gripper pose, the pre-grasping height, etc., and the action representation is realized through discretization or continuous space parameterization. A multi-objective reward function is designed by combining the grasping stability (such as contact area, friction torque), error tolerance (the influence of end error on grasping success rate), and motion feasibility (obstacle avoidance constraint) to guide the reinforcement learning agent to explore high-robustness grasping strategies. Deep Q network (DQN) or proximal policy optimization (PPO) and other reinforcement learning algorithms are used, the error transmission model is used as an environment simulator, and the candidate scheme set containing different grasping points and poses is generated by interacting with the virtual environment.

[0051] S04, based on the preset priority evaluation rule, the candidate set of grabbing scheme is comprehensively scored and ranked, and the scheme with the highest comprehensive score is selected as the optimal grabbing scheme.

[0052] The preset priority evaluation rule can include the following evaluation indexes: grabbing stability: based on the consistency of the normal vector of the grabbing point, the contact area between the gripper and the object surface, the friction torque and other geometric parameters, the anti-disturbance ability during grabbing is evaluated; error tolerance: combined with the error transmission model of Monte Carlo simulation, the influence degree of the end effector pose deviation on the success rate of grabbing is quantified; motion feasibility: detect whether there is collision risk in the motion trajectory of the robot arm from the current pose to the grabbing pose, evaluate whether the joint angle exceeds the physical limit; work efficiency: calculate the indicators such as the length of the grabbing path and the energy consumption of joint motion, and optimize the work efficiency in industrial scene; environmental adaptability: for interference factors such as light and vibration, the robustness of the scheme in actual working condition is evaluated. Using analytic hierarchy process (AHP) or dynamic weight adjustment strategy, according to different work scenes (such as precision assembly, heavy lifting), the priority weight coefficients of each index are preset, the comprehensive score of the candidate scheme is calculated by using linear weighted model, and the comprehensive scores of all candidate schemes are ranked in descending order, the scheme with the highest score is selected as the optimal grabbing scheme, and the score details of each scheme are output for system debugging reference.

[0053] S05, according to the parameter characteristics of the optimal grabbing scheme, the joint motion trajectory of the robot arm and the corresponding control instruction set are generated.

[0054] The joint space mapping based on kinematic inverse solution can be used to establish a kinematic inverse solution model of the robot arm, with the end position (three-dimensional coordinates + attitude matrix) of the optimal grasping scheme as the input. An improved Levenberg-Marquardt (an iterative optimization algorithm for solving nonlinear least squares problems) iterative algorithm is adopted, combined with the physical constraints of the joints of the robot arm (angle range, speed limit), to solve the joint angle combination that meets the end position. For the multi-solution problem of the redundant degree of freedom robot arm, the optimal joint solution is selected based on the following principles: the joint angle is close to the middle position, avoiding reaching the physical limit; the joint motion range is minimized, reducing energy consumption; combined with the error transfer model, the joint combination with the lowest sensitivity to end error is selected. The grasping process can be divided into three stages: Approach (approach), Grasp (grasp), and Withdraw (withdraw). The joint angle, speed, and acceleration of each stage are interpolated by a quintic polynomial, and the endpoint constraints (position, speed, and acceleration) are solved to generate the joint motion trajectory of the robot arm. Meanwhile, the dynamics model of the robot arm can be introduced to dynamically smooth the trajectory: limit the joint jerk to reduce motion impact; based on the trajectory error feedback of the end effector, the polynomial parameters are adjusted in real time to compensate for modeling errors. The trajectory is discretized into a time series of joint angle commands, and a command buffer queue is constructed. Real-time control is achieved through the following strategies: based on the deviation of the current position of the robot arm from the target trajectory, a proportional-integral-derivative (PID) controller is used to dynamically adjust the joint output; integrate force feedback signals (such as grasping force sensor data) to trigger trajectory re-planning when abnormal contact is detected, avoiding collision or grasping failure.

[0055] The above-mentioned robot arm positioning and grasping method based on machine vision uses a multi-view camera array to synchronously acquire RGB-D images of the target scene and fuse them to generate three-dimensional point cloud data. The improved LSD algorithm and PnP algorithm are used to suppress feature matching deviations caused by local occlusion and texture loss, reducing the initial pose estimation error. The generative adversarial network is used to eliminate light distortion to improve the surface reflectivity reconstruction accuracy. The Monte Carlo simulation is used to construct a robot motion error transfer model, and the reinforcement learning is used to generate candidate grasping schemes. The optimal scheme is selected through a pre-set priority evaluation rule, and the robot arm joint motion trajectory and control instruction set are generated. This method effectively solves the problems of large target positioning deviation and light interference sensitivity in complex environments, improves the grasping accuracy and robustness of the robot arm, and can adapt to the needs of high anti-interference positioning and grasping in complex scenes such as industrial sorting and flexible assembly.

[0056] In one embodiment, the RGB-D image of the target scene is processed by data fusion to generate three-dimensional point cloud data, including:

[0057] S11, a mapping relationship of the RGB-D image of the target scene is established, an image registration algorithm is used to align the RGB-D image, and an aligned image is obtained;

[0058] S12, a geometric segmentation method based on point cloud normal vector and curvature analysis is used to segment the target object in the aligned image, and a segmented image is obtained;

[0059] S13, a local binary pattern is used to extract texture features of the target object in the segmented image, and a texture feature vector is obtained;

[0060] S14, the geometric boundary point coordinates of the segmented image and the corresponding texture feature vector are spatiotemporally synchronized and coded, and a three-dimensional point cloud structured dataset containing geometric, texture and timestamp information is fused and generated.

[0061] Specifically, the specific process of generating three-dimensional point cloud data from the RGB-D image of the target scene through data fusion processing is as follows: a mapping relationship between the RGB image and the depth image is established, an image registration algorithm is used to eliminate the perspective deviation to realize the spatial alignment of the multi-modal data, and an aligned image is obtained; a geometric segmentation method based on point cloud normal vector and curvature analysis is used to segment the target object from the background by calculating the local normal vector distribution and curvature change characteristics of the point cloud, and a segmented image is obtained; a local binary pattern (LBP) is used to extract texture features of the target object in the segmented image, and a feature vector representing surface texture information is generated; the three-dimensional coordinates of the geometric boundary points of the segmented image and the corresponding texture feature vector are spatiotemporally synchronized and coded, and a three-dimensional point cloud structured dataset containing object geometric structure, surface texture and timestamp information is fused and generated, providing multi-dimensional data support with spatial precision and texture features for subsequent pose calculation.

[0062] In one embodiment, an improved LSD algorithm and PnP algorithm are used to calculate the initial pose of the target object in the three-dimensional point cloud data, including:

[0063] S21, based on the anisotropic diffusion equation corrected by the Gaussian kernel function, a multi-scale line feature detection model is constructed, the three-dimensional point cloud data is iteratively filtered, and a geometric feature point set of the target object contour is generated;

[0064] S22, according to the geometric feature point set, a random sampling sequence based on RANSAC is constructed in the PnP algorithm solving framework, and the initial solution of the pose parameters is calculated by minimizing the re-projection error function;

[0065] S23, the initial solution of the pose parameters is input into the Levenberg-Marquardt optimizer, the rotation matrix and the translation vector are nonlinearly iteratively solved, and the initial pose containing six degrees of freedom is output.

[0066] Specifically, in this embodiment, the process of calculating the initial pose of the target object by using the improved LSD algorithm and the PnP algorithm is as follows: based on the Gaussian kernel function, the anisotropic diffusion equation is modified, a multi-scale line feature detection model is constructed, the three-dimensional point cloud data is iteratively filtered, the geometric features of the target object contour are retained while the noise is suppressed, and a precise contour feature point set is generated; based on the feature point set, a random sampling sequence based on RANSAC (Random Sample Consensus) is constructed in the PnP algorithm solving framework, the re-projection error function containing Geman-McClure (a robust kernel function) is minimized, the initial solution of the pose parameters, i.e., the rotation matrix and the translation vector, is calculated, and the interference of abnormal feature points is eliminated; the initial solution is input into the Levenberg-Marquardt optimizer, the rotation matrix and the translation vector are nonlinearly iteratively optimized, and the six-degree-of-freedom initial pose containing three-dimensional translation and three-dimensional rotation is output, thereby realizing high-precision estimation of the pose of the target object.

[0067] In one of the embodiments, the illumination distortion elimination processing is performed in combination with the generative adversarial network to generate the three-dimensional coordinates of the target object, which includes:

[0068] S31, a residual learning model based on the conditional generative adversarial network is constructed, and the illumination-invariant texture feature vector is generated through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data;

[0069] S32, the illumination-invariant texture feature vector and the initial pose parameter are input into the conditional generator, the adversarial loss function is constructed through the Markov discriminator, and the object surface reflectance map without illumination distortion is iteratively generated;

[0070] S33, based on the pixel consistency constraint of the object surface reflectance map, the multi-view triangulation reconstruction is performed on the three-dimensional point cloud data, and the three-dimensional coordinates of the target object containing the geometric topological structure are generated.

[0071] Exemplarily, the process of illumination distortion elimination and three-dimensional coordinate generation in combination with a generative adversarial network is as follows: a residual learning model based on a conditional generative adversarial network (cGAN) is constructed, the influence of illumination change on surface reflection is stripped by analyzing the normal vector distribution of a three-dimensional point cloud geometric feature point set, and an illumination-independent texture feature vector is generated; the illumination-invariant texture feature vector and initial pose parameters are input into a conditional generator, an adversarial loss function is constructed by using a Markov discriminator, and an object surface reflectance map free of illumination distortion is generated by iterative game between the generator and the discriminator, so that the real reflection characteristics of the object are restored; based on the pixel consistency constraint of the reflectance map, multi-view point cloud data is subjected to triangulation reconstruction, a three-dimensional coordinate of the target object containing geometric topological structure is generated, and high-precision three-dimensional positioning under a complex illumination environment is realized.

[0072] In one embodiment, S41, an initial solution of the pose parameters is calculated by minimizing a re-projection error function, using the following energy function model:

[0073]

[0074] wherein R∈SO(3) represents a rotation matrix, represents a translation vector, represents a camera perspective projection operator, and π([x, y, z] T ) = [f x x / z + c x , f y y / z + c y ] T , f x and f y represent focal lengths, c x and c y represent principal points, represents a camera intrinsic matrix, represents the i-th geometric feature point coordinate segmented from a three-dimensional point cloud structured data set, x i ∈R 2 represents a two-dimensional observation point coordinate matched with X i , and represents a Geman-McClure robust kernel function.

[0075] Specifically, the formula constructs an optimization objective function of the pose parameters (rotation matrix (R), translation vector (t)) by quantifying the projection error of the three-dimensional point cloud feature points in the camera imaging plane. The rotation matrix R∈SO(3) represents the rotation attitude of the target object in the three-dimensional space, the translation vector represents the position offset of the target object in the camera coordinate system, [R|t] constitutes a camera extrinsic matrix, realizes the transformation from the world coordinate system to the camera coordinate system, and the camera intrinsic matrix focal length f x and f y and the principal coordinate point (c x , c y ) define the imaging projection relationship. The projection operator π projects a three-dimensional space point X i to a two-dimensional image plane to obtain a theoretical projection point π(K[R|t]X i ). The Euclidean distance square of the two-dimensional observation point coordinate x i observed in practice and the theoretical projection point is That is, the re-projection error of a single feature point. The Geman-McClure kernel function is defined as When the error s is small, approximately linear weighting, retaining the error contribution of effective feature points; when the error s is large, ρ(s)→1, the weight is saturated, and the influence of abnormal points (such as mis-matched features, noise points) is suppressed. In the PnP algorithm solving framework, the kernel function is combined with RANSAC (Random Sample Consensus): RANSAC generates candidate pose solutions by randomly sampling three-dimensional-two-dimensional point pairs; the error of each feature point is weighted using ρ(s), and the global energy function E(R, t) is calculated; the pose solution that minimizes E(R, t) is iteratively screened, and the feature points judged as abnormal by ρ(s) are removed, realizing the joint optimization of pose estimation-abnormal point filtering.

[0076] In one of the embodiments, S51, a multi-scale line feature detection model is constructed to perform iterative filtering processing on the three-dimensional point cloud data, and the following anisotropic diffusion equation is used:

[0077]

[0078] wherein I(x, y, t) represents the image intensity field after Gaussian kernel ρ smoothing, p ∈ [0.5, 1.5], denotes a diffusion coefficient function based on the Weickert tensor correction, σ represents an edge preservation threshold, and α ∈ [1.5, 2.0] represents an anisotropy index, denotes the gradient field calculated after Gaussian smoothing of the image intensity field with a scale length of p, and the iteration termination condition of the diffusion equation is k represents the number of iterations, and ||·|| F denotes the Frobenius norm.

[0079] Exemplarily, the equation realizes multi-scale filtering of the three-dimensional point cloud through a gradient-driven diffusion process. For the diffusion direction, diffusion is performed along the gradient vertical direction of the image intensity field I, and edge features are retained. For the diffusion intensity adjustment, the diffusion coefficient function g σAdaptive adjustment of diffusion amount, to achieve the balance of edge protection-noise suppression. Its adaptive diffusion process is: for the smoothing stage: when g σ (s)≈1, the diffusion intensity is large, and the random noise is effectively suppressed; for the edge preservation stage: when g σ (s)≈0, the diffusion is suppressed, and the edge definition is maintained. By adjusting p∈[0.5, 1.5] in the range of iteration calculation, the following is realized: fine scale (p→0.5), capturing fine structure and sharp edges; coarse scale (p→1.5), extracting overall contour and main structure; through the superposition of gradient fields of different scales, a point cloud representation containing multi-resolution line features is generated. The iteration termination condition of the formula is to ensure that the diffusion process is terminated when the intensity field changes less than 10 -3 By fusing the anisotropic diffusion theory in mathematics and the geometric features of point cloud in physics, the feature mismatching problem of traditional methods in texture missing and light change scenes is solved.

[0080] In one of the embodiments, S61, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50cm, 100cm], and each module synchronously collects RGB-D images at a frame rate of 60Hz. The data fusion processing adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3mm.

[0081] Specifically, 8 binocular vision modules are arranged in a ring shape with a radius range of 50 cm to 100 cm. The technical advantages are as follows: 8 modules form a 360° ring field of view, can synchronously collect RGB-D data of each orientation of the target object, solve the problem of shielding blind area of traditional single / dual binocular camera, and are especially suitable for full-orientation modeling of complex posture objects; the radius adjustable design (50 cm to 100 cm) is suitable for different sizes of targets (such as small parts to medium workpieces), and by adjusting the array radius, the spatial resolution and the field of view range can be balanced. 60 Hz high frame rate can effectively suppress image blur caused by mechanical arm movement or target object shaking; through the hardware trigger synchronization mechanism, the timestamp deviation of multi-view data can be ensured to be less than or equal to 1 ms, which provides an accurate space-time reference for subsequent point cloud fusion, and avoids three-dimensional reconstruction distortion caused by time misalignment. A spatial downsampling strategy based on voxel hashing is adopted, the point cloud resolution is set to 3 mm, the original point cloud (usually containing 1 million + points) is compressed to 100,000 levels through voxelization, the calculation efficiency is improved by 80%, and the geometric features such as 3 mm voxel are retained. The retention rate of edge points with curvature change rate > 0.1 rad / mm is 92%; resolution adaptation: 3 mm resolution takes into account industrial grasping accuracy (end position error of mechanical arm ≤ 0.5 mm) and calculation resource consumption. The object contour error of point cloud reconstruction under this resolution is ≤ 0.8 mm, which can meet the demand of precise assembly scene.

[0082] The above-mentioned mechanical arm positioning and grabbing method based on machine vision uses a multi-view camera array to synchronously acquire an RGB-D image of a target scene and generates a three-dimensional point cloud structured data set containing geometric, texture and timestamp information through data fusion, constructs a multi-scale line feature detection model based on the anisotropic diffusion equation modified based on the Gaussian kernel function in the improved LSD algorithm to iteratively filter and generate a geometric feature point set of the target object contour, calculates an initial solution of the pose parameters based on the random sampling sequence and Geman-McClure robust kernel function minimum re-projection error in the PnP algorithm framework, outputs an initial six-degree-of-freedom pose with a Levenberg-Marquardt optimizer, generates a texture feature vector that is invariant to illumination and iteratively generates an object surface reflectance map that is free of illumination distortion through a conditional generative adversarial network residual learning model, reconstructs the three-dimensional coordinates of the target object based on pixel consistency constraints, constructs a mechanical arm motion error transfer model based on Monte Carlo simulation and generates a candidate grasping scheme set in combination with reinforcement learning, filters the optimal scheme through a preset priority evaluation rule and generates a joint motion trajectory and control instruction set, effectively solving the technical problems of large target positioning deviation and illumination interference sensitivity in a complex environment, suppressing feature matching deviation caused by local occlusion and texture missing through multi-modal data fusion and nonlinear optimization algorithm, eliminating the influence of illumination distortion on three-dimensional reconstruction, improving the mechanical arm grasping precision and robustness in combination with error transfer modeling and reinforcement learning decision-making, and realizing high success rate positioning and grabbing in industrial sorting, flexible assembly and other scenes.

[0083] It should be understood that although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0084] Based on the same inventive concept, the embodiment of the present application also provides a machine vision-based robotic arm positioning and grabbing device for implementing the machine vision-based robotic arm positioning and grabbing method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more machine vision-based robotic arm positioning and grabbing device embodiments provided below can refer to the limitations of the machine vision-based robotic arm positioning and grabbing method described above, which will not be repeated here.

[0085] In one exemplary embodiment, as shown in Figure 2 A machine vision-based robotic arm positioning and grabbing device is provided, comprising:

[0086] The point cloud reconstruction module 101 is configured to synchronously acquire RGB-D images of a target scene by using a multi-view camera array, and generate three-dimensional point cloud data by data fusion processing;

[0087] The pose optimization module 102 is configured to calculate an initial pose of a target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and perform light distortion elimination processing by combining a generative adversarial network to generate a three-dimensional coordinate of the target object;

[0088] The error propagation module 103 is configured to construct a robotic arm motion error propagation model by Monte Carlo simulation based on the three-dimensional coordinate, and generate a candidate grabbing scheme set by combining a reinforcement learning algorithm;

[0089] The grabbing decision module 104 is configured to comprehensively score and sort the candidate grabbing scheme set based on a preset priority evaluation rule, and select a scheme with the highest comprehensive score as an optimal grabbing scheme;

[0090] The trajectory planning module 105 is configured to generate a robotic arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grabbing scheme.

[0091] In one embodiment, the point cloud reconstruction module 101 is further configured to:

[0092] Establish a mapping relationship of the RGB-D images of the target scene, align the RGB-D images by using an image registration algorithm, and obtain aligned images;

[0093] Segment the target object in the aligned images by using a geometric segmentation method based on point cloud normal vector and curvature analysis, and obtain segmented images;

[0094] Extract texture features of the target object in the segmented images by using a local binary pattern, and obtain a texture feature vector;

[0095] The geometric boundary point coordinates of the segmented image are spatiotemporally synchronized with corresponding texture feature vectors for fusion to generate a three-dimensional point cloud structured dataset containing geometric, texture and timestamp information.

[0096] In one of the embodiments, the pose optimization module 102 is further configured to:

[0097] Based on the anisotropic diffusion equation modified by the Gaussian kernel function, a multi-scale line feature detection model is constructed to perform iterative filtering on the three-dimensional point cloud data to generate a geometric feature point set of the object contour.

[0098] According to the geometric feature point set, a random sampling sequence based on RANSAC is constructed in the PnP algorithm solving framework to calculate an initial solution of the pose parameters by minimizing a re-projection error function.

[0099] The initial solution of the pose parameters is input into a Levenberg-Marquardt optimizer to perform nonlinear iterative solving on the rotation matrix and the translation vector to output an initial pose containing six degrees of freedom.

[0100] In one of the embodiments, the pose optimization module 102 is further configured to:

[0101] A residual learning model based on a conditional generative adversarial network is constructed to generate illumination-invariant texture feature vectors from the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data.

[0102] The illumination-invariant texture feature vectors and the initial pose parameters are input into a conditional generator to construct an adversarial loss function through a Markov discriminator to iteratively generate an object surface reflectance map free from illumination distortion.

[0103] Based on the pixel consistency constraint of the object surface reflectance map, multi-view triangulation reconstruction is performed on the three-dimensional point cloud data to generate a target object three-dimensional coordinate containing geometric topological structure.

[0104] In one of the embodiments, the pose optimization module 102 is further configured to calculate the initial solution of the pose parameters using the following energy function model:

[0105]

[0106] wherein R∈SO(3) represents a rotation matrix, represents a translation vector, represents a camera perspective projection operator, and π([x, y, z] T ) = [f x x / z + c x , f y y / z + c y ] T , f xand f y denotes focal length, c x and c y denotes the principal coordinate point, denotes the camera intrinsic matrix, denotes the i-th geometric feature point coordinate segmented from the three-dimensional point cloud structured data, x i ∈R 2 denotes the RGB-D image and X i matched two-dimensional observation point coordinate, denotes the Geman-McClure robust kernel function.

[0107] In one of the embodiments, the pose optimization module 102 is further configured to perform iterative filtering processing on the three-dimensional point cloud data using the following anisotropic diffusion equation:

[0108]

[0109] wherein I(x, y, t) denotes the image intensity field smoothed by the Gaussian kernel p, p ∈ [0.5, 1.5], denotes the diffusion coefficient function based on the Weickert tensor correction, σ denotes the edge preservation threshold, and a ∈ [1.5, 2.0] denotes the anisotropy index, denotes the gradient field calculated after the image intensity field is Gaussian smoothed with a scale length of p, and the iteration termination condition of the diffusion equation is k denotes the iteration number, and ||·|| F denotes the Frobenius norm.

[0110] In one of the embodiments, the multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50 cm, 100 cm], each module synchronously collects RGB-D images at a frame rate of 60 Hz, and the data fusion processing adopts a spatial downsampling strategy based on voxel hashing, and the point cloud resolution is set to 3 mm.

[0111] In one of the embodiments, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the machine vision-based mechanical arm positioning and grabbing method as described above when executing the computer program.

[0112] In one of the embodiments, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above method embodiments.

[0113] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only schematic, wherein the components shown as separate components can or can not be physically separate, and the components shown as a unit can or can not be a physical unit, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure. Those skilled in the art can understand and implement it without creative labor.

[0114] The above-described embodiments only express several implementation manners of the present application, which are described in detail, but cannot be understood as a limitation on the patent scope of the application. It should be pointed out that, for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A machine vision-based robotic arm positioning and grasping method, characterized by, The method comprises: synchronously acquiring an RGB-D image of a target scene by using a multi-view camera array, and generating three-dimensional point cloud data by data fusion processing; calculating the initial pose of the target object in the three-dimensional point cloud data by using an improved LSD algorithm and a PnP algorithm, and eliminating light distortion by combining a generative adversarial network to generate three-dimensional coordinates of the target object; based on the three-dimensional coordinates, constructing a mechanical arm motion error transfer model by Monte Carlo simulation, and generating a candidate grasping scheme set by combining a reinforcement learning algorithm; comprehensively scoring and ranking the candidate grasping scheme set based on a preset priority evaluation rule, and selecting the scheme with the highest comprehensive score as the optimal grasping scheme; generating a mechanical arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme; wherein the calculation of the initial pose of the target object in the three-dimensional point cloud data by using the improved LSD algorithm and the PnP algorithm comprises: based on a Gaussian kernel function to correct an anisotropic diffusion equation, constructing a multi-scale line feature detection model, iteratively filtering the three-dimensional point cloud data, and generating a geometric feature point set of the target object contour; according to the geometric feature point set, constructing a random sampling sequence based on RANSAC in a PnP algorithm solving framework, and calculating an initial solution of the pose parameters by minimizing a re-projection error function; inputting the initial solution of the pose parameters into a Levenberg-Marquardt optimizer to nonlinearly iteratively solve the rotation matrix and the translation vector, and outputting an initial pose containing six degrees of freedom; wherein the elimination of light distortion by combining the generative adversarial network comprises: constructing a residual learning model based on a conditional generative adversarial network, generating a light-invariant texture feature vector through the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data; inputting the light-invariant texture feature vector and the initial pose parameters into a conditional generator, constructing an adversarial loss function through a Markov discriminator, and iteratively generating an object surface reflectance map without light distortion; based on the pixel consistency constraint of the object surface reflectance map, performing multi-view triangulation reconstruction on the three-dimensional point cloud data to generate a target object three-dimensional coordinate containing a geometric topology structure.

2. The method of claim 1, wherein, The RGB-D image of the target scene is processed by data fusion to generate three-dimensional point cloud data, comprising: establishing a mapping relationship of the RGB-D image of the target scene, aligning the RGB-D image by using an image registration algorithm to obtain an aligned image; segmenting the target object in the aligned image by using a geometric segmentation method based on point cloud normal vector and curvature analysis to obtain a segmented image; performing texture feature extraction processing on the target object in the segmented image by using a local binary pattern to obtain a texture feature vector; spatiotemporally synchronously encoding the geometric boundary point coordinates of the segmented image and the corresponding texture feature vector to fuse and generate a three-dimensional point cloud structured data set containing geometric, texture and timestamp information.

3. The method of claim 1, wherein, The initial solution of the pose parameter is calculated by minimizing the re-projection error function, using the following energy function model: where R ∈ SO(3) denotes a rotation matrix, denotes a translation vector, denotes a camera perspective projection operator, and π([x, y, z] T ) = [f x x / z + c x , f y y / z + c y ] T , f x and f y denote focal lengths, c x and c y denote principle points, denotes a camera intrinsic matrix, denotes the i-th geometric feature point coordinate segmented from the 3D point cloud structured data, x i ∈ R 2 denotes the 2D observation point coordinate matched with X i in the RGB-D image, denotes the Geman-McClure robust kernel function.

4. The method of claim 1, wherein, The multi-scale line feature detection model is used for iterative filtering of the three-dimensional point cloud data, using the following anisotropic diffusion equation: where I(x, y, t) represents the image intensity field after Gaussian kernel p smoothing, p e [0.5, 1.5], represents the diffusion coefficient function based on Weickert tensor modification, σ represents the edge preservation threshold, a e [1.5, 2.0] represents the anisotropy index, represents the gradient field calculated after Gaussian smoothing with scale length p on the image intensity field, and the iteration termination condition of the diffusion equation is k represents the iteration number, and ||·|| F represents the Frobenius norm. F represents the Frobenius norm.

5. The method of claim 1, wherein, The multi-view camera array is composed of 8 binocular vision modules arranged in a ring with a radius of [50cm, 100cm], each module synchronously collects RGB-D images at a frame rate of 60Hz, and the data fusion processing adopts a spatial downsampling strategy based on voxel hashing, with a point cloud resolution of 3mm.

6. A machine vision based robotic arm positioning and grasping device for implementing the method of any one of claims 1 to 5, characterized in that, The device comprises: a point cloud reconstruction module for synchronously acquiring RGB-D images of a target scene using a multi-view camera array, and generating three-dimensional point cloud data through data fusion processing; a pose optimization module for calculating an initial pose of a target object in the three-dimensional point cloud data using an improved LSD algorithm and a PnP algorithm, and performing illumination distortion elimination processing using a generative adversarial network to generate a three-dimensional coordinate of the target object; an error transfer module for constructing a mechanical arm motion error transfer model based on the three-dimensional coordinate through Monte Carlo simulation, and generating a candidate grasping scheme set using a reinforcement learning algorithm; a grasping decision module for comprehensively scoring and sorting the candidate grasping scheme set based on a preset priority evaluation rule, and selecting the scheme with the highest comprehensive score as the optimal grasping scheme; a trajectory planning module for generating a mechanical arm joint motion trajectory and a corresponding control instruction set according to the parameter characteristics of the optimal grasping scheme; wherein the pose optimization module is further configured to: correct the anisotropic diffusion equation based on a Gaussian kernel function, construct a multi-scale line feature detection model, and perform iterative filtering of the three-dimensional point cloud data to generate a geometric feature point set of the target object contour; construct a random sampling sequence based on RANSAC in a PnP algorithm solving framework, and calculate an initial solution of the pose parameter by minimizing a re-projection error function; input the initial solution of the pose parameter into a Levenberg-Marquardt optimizer, perform nonlinear iterative solving on the rotation matrix and translation vector, and output an initial pose containing six degrees of freedom; wherein the pose optimization module is further configured to: construct a residual learning model based on a conditional generative adversarial network, generate an illumination-invariant texture feature vector based on the normal vector distribution of the geometric feature point set in the three-dimensional point cloud data; input the illumination-invariant texture feature vector and the initial pose parameter into a conditional generator, construct an adversarial loss function through a Markov discriminator, and iteratively generate an object surface reflectance map without illumination distortion; perform multi-view triangulation reconstruction on the three-dimensional point cloud data based on the pixel consistency constraint of the object surface reflectance map, and generate a target object three-dimensional coordinate containing geometric topology structure. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images