A vision-adaptive guidance method and system for robotic arms based on 3D point clouds

Abstract

This invention discloses a vision-adaptive guidance method and system for a robotic arm based on 3D point clouds, belonging to the field of robot intelligent control. Addressing the problem that existing fixed guidance strategies cannot adapt to dynamic changes in working conditions, this invention adds a working condition perception step between point cloud acquisition and feature extraction: calculating the density distribution uniformity index, noise level quantization index, and surface normal consistency index of each frame of point cloud data, combining them into a three-dimensional feature vector; determining the current working condition level based on this feature vector; selecting a corresponding guidance parameter set from a preset mapping relationship according to the working condition level, this parameter set at least includes the filter kernel size, feature extraction sampling rate, pose estimation iteration count, and convergence threshold; finally, performing point cloud preprocessing, feature extraction, and pose estimation using the selected parameter set. This invention enables real-time perception and dynamic adjustment of the guidance strategy to changes in working conditions, improving the success rate of the robotic arm's grasping in complex scenarios and the system response efficiency.

Description

Technical Field

[0001] This application belongs to the technical field of robot intelligent control, specifically relating to a vision-adaptive guidance method and system for a robotic arm based on 3D point clouds. Background Technology

[0002] Machine vision technology, as one of the key enabling technologies for intelligent manufacturing, has been widely applied in various aspects of industrial automation. Among them, 3D point cloud-based robotic arm vision guidance technology, by acquiring the three-dimensional spatial information of the target object, enables precise guidance of the robotic arm's grasping or placing actions, playing a crucial role in scenarios such as material sorting, precision assembly, and logistics handling. 3D point cloud data provides rich spatial structural information and, compared to two-dimensional images, has stronger resistance to lighting interference and more accurate spatial positioning capabilities, thus becoming the preferred data source for high-precision vision guidance systems.

[0003] Among them, the 3D point cloud-based robotic arm vision guidance method mainly involves core steps such as point cloud acquisition, feature extraction, pose estimation, and path planning. By extracting the geometric features of the target object from the point cloud data and calculating its three-dimensional pose, it provides the robotic arm with accurate grasping guidance information. The basic process of this method is as follows: first, point cloud data of the target scene is acquired using a 3D sensor; then, the point cloud is preprocessed and features are extracted; next, the pose of the target object is estimated based on feature matching or learning methods; finally, the motion trajectory of the robotic arm is planned based on the pose information, and the grasping operation is executed.

[0004] In the prior art, such as the 3D machine vision-guided manipulator method and device disclosed in CN111152229B, and the three-dimensional vision-guided manipulator grasping method and system disclosed in CN109927036A, in the system initialization stage, the point cloud preprocessing parameters, feature extraction algorithms and pose estimation strategies are determined according to the preset typical working conditions, and remain unchanged throughout the entire working cycle.

[0005] However, in actual industrial production environments, operating parameters such as changes in lighting conditions, differences in object stacking states, and varying degrees of target object occlusion are constantly changing. When the operating conditions deviate from the initial settings, the adaptability of the fixed guidance strategy decreases significantly.

[0006] Specifically, when the complexity of the working conditions increases, the preset feature extraction sampling rate and pose estimation iteration number may not be sufficient to cope with noise interference and occlusion problems, resulting in reduced pose estimation accuracy or even failure of guidance; while when the working conditions are relatively simple, excessively high processing parameters will waste computing resources and affect the system response speed.

[0007] Therefore, how to achieve real-time perception and dynamic adjustment of guidance strategies to changes in working conditions, so that the robot vision guidance system can automatically select the optimal guidance parameters and strategies according to the current working conditions, has become a key problem that urgently needs to be solved in current technical research. Summary of the Invention

[0008] The purpose of this invention is to provide a vision-adaptive guidance method and system for robotic arms based on 3D point clouds, which can effectively solve the problem in the background art where fixed guidance strategies cannot adapt to changes in working conditions, resulting in decreased guidance accuracy or guidance failure.

[0009] To achieve the above objectives, the technical solution adopted by this invention is: a visual adaptive guidance method for a robotic arm based on 3D point clouds, comprising: The steps for acquiring 3D point cloud data of a target scene, and the steps for feature extraction and pose estimation of the 3D point cloud data of the target scene; Also includes: Between the point cloud acquisition step and the feature extraction step, the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index are calculated for each frame of point cloud data and combined into a three-dimensional feature vector. The current operating condition level is determined based on the three-dimensional feature vector. Based on the operating condition level, a corresponding set of guiding parameters is selected from a preset mapping relationship. The set of guiding parameters includes at least the point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number, and convergence threshold. Point cloud preprocessing, feature extraction, and pose estimation are performed using the selected set of guiding parameters.

[0010] Furthermore, the density distribution uniformity index is calculated as follows: The point cloud data space is divided into a three-dimensional voxel mesh, and the number of voxels is statistically analyzed. The number of point clouds for individual elements is The total number of voxels is ; Calculate the arithmetic mean of the number of voxel point clouds. ; Calculate the standard deviation of the number of point clouds for each voxel. ; The ratio of standard deviation to mean is defined as the density distribution uniformity index. .

[0011] Furthermore, the point cloud noise level quantization index is calculated as follows: For each sampling point in the point cloud Retrieve the sampling points The closest in three-dimensional space 1 neighboring point; Calculate the sampling points With the Local variance of Euclidean distance between neighboring points ,in For the first The sampling point and the first The sampling point of the first sampling point Euclidean distance between neighboring points For the sampling point With the The arithmetic mean of the Euclidean distances between the nearest neighbors. ; Local variance over all sampling points The global average value is used as the quantitative indicator of point cloud noise level. , This represents the total number of sampling points in the point cloud.

[0012] Furthermore, the surface normal consistency index is calculated as follows: For each sampling point in the point cloud Select the sampling points The closest in three-dimensional space A neighborhood set is formed by a number of neighboring points; Constructing the covariance matrix ,in Indicates the sampling point The first in the neighborhood set Three-dimensional spatial coordinate vectors of neighboring points Let be the centroid coordinate vector of the neighborhood set; For the covariance matrix Eigenvalue decomposition yields three eigenvalues. , , Sort in descending order; Calculate the global surface normal consistency index , This represents the total number of sampling points in the point cloud.

[0013] Furthermore, the determination of the current operating condition level adopts a threshold-based hierarchical decision-making method: Set a first threshold and a second threshold for each of the three feature quantities, wherein the three feature quantities are density distribution uniformity indices. Point cloud noise level quantification index and surface normal consistency index ; When density distribution uniformity index Less than or equal to the density distribution uniformity index The corresponding first threshold And point cloud noise level quantification index Less than or equal to the point cloud noise level quantization index The corresponding first threshold And the surface normal consistency index Greater than or equal to the surface normal consistency index The corresponding first threshold At that time, it was determined to be a simple working condition; When density distribution uniformity index Less than or equal to the density distribution uniformity index The corresponding second threshold And point cloud noise level quantification index Less than or equal to the point cloud noise level quantization index The corresponding second threshold And the surface normal consistency index Greater than or equal to the surface normal consistency index The corresponding second threshold At that time, it was determined to be a normal working condition; When density distribution uniformity index Greater than the density distribution uniformity index The corresponding second threshold or point cloud noise level quantification index Greater than the point cloud noise level quantization index The corresponding second threshold Or surface normal consistency index Smaller than the surface normal consistency index The corresponding second threshold At that time, it was determined to be a complex working condition.

[0014] Furthermore, the determination of the current operating condition level uses a machine learning classifier; The machine learning classifier is a three-layer fully connected feedforward network. The input layer of the three-layer fully connected feedforward network receives the three-dimensional feature vector, and the hidden layers of the three-layer fully connected feedforward network contain... The output layer of the three-layer fully connected feedforward network contains nodes and uses a linear rectified unit activation function. The output layer outputs three level labels—simple working condition label, general working condition label, and complex working condition label—using the Softmax activation function. The category with the highest probability value is used as the working condition level label output by the machine learning classifier.

[0015] Furthermore, the guiding parameters are set as follows: The guiding parameter set corresponding to the simple working condition is: the point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice; The typical operating condition's corresponding guidance parameter set is: Point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice; The guidance parameter set corresponding to complex working conditions is: the point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice.

[0016] Furthermore, during the process of the robotic arm generating motion trajectories, performing gripping actions, transporting materials to the unloading position, and releasing them, the speed is [per second]. Frame to The frame rate is used to continuously collect point cloud data of the target scene, and to extract working condition features and determine the working condition level. When a change in operating condition level is detected, a guided parameter set switching process is triggered. This process uses linear interpolation to gradually transition the parameter values ​​of the current parameter set to the parameter values ​​corresponding to the new level. The total transition time is set to [missing information]. Instant Second.

[0017] Furthermore, the steps of calculating the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index are performed in the working condition sensing unit. The working condition perception unit adopts a parallel computing architecture, which is as follows: the point cloud data is divided into several data blocks according to spatial regions, each data block is assigned an independent computing thread, each thread executes the corresponding feature calculation task in parallel, and the results of each data block are summarized and output after the calculation is completed.

[0018] Furthermore, A vision-adaptive guidance system for robotic arms based on 3D point clouds, comprising: A point cloud acquisition module, which is used to acquire three-dimensional point cloud data of the target scene; The working condition perception unit is deployed between the point cloud acquisition module and the feature extraction module. The working condition perception unit receives each frame of point cloud data output by the point cloud acquisition module. The working condition perception unit is used to calculate the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index for each frame of point cloud data, and combine them into a three-dimensional feature vector. The working condition classification model module receives the three-dimensional feature vector output by the working condition sensing unit, and is used to determine the current working condition level based on the three-dimensional feature vector. The adaptive selection module for guiding strategy parameters receives the working condition level label output by the working condition classification model module. The adaptive selection module for guiding strategy parameters is used to select the corresponding set of guiding parameters from a preset mapping relationship based on the working condition level. The set of guiding parameters includes at least the point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number and convergence threshold. The feature extraction module receives the point cloud preprocessing filter kernel size and feature extraction sampling rate output by the guidance strategy parameter adaptive selection module. The feature extraction module is used to perform point cloud preprocessing and feature extraction using the point cloud preprocessing filter kernel size and the feature extraction sampling rate. The pose estimation module receives the pose estimation iteration number and convergence threshold output by the guidance strategy parameter adaptive selection module. The pose estimation module is used to perform pose estimation using the pose estimation iteration number and the convergence threshold, and outputs the three-dimensional pose information of the target object. The trajectory planning and execution module receives the three-dimensional pose information output by the pose estimation module, and is used to plan the motion trajectory of the robot arm and perform the grasping operation based on the three-dimensional pose information.

[0019] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention adds a working condition sensing unit after the 3D point cloud acquisition module to analyze the density distribution characteristics, noise characteristics and surface normal consistency characteristics of point cloud data in real time, so as to realize the real-time perception of changes in the working environment. It upgrades the traditional static guidance strategy into a closed-loop system of dynamic perception and adjustment, effectively solving the technical problem of poor adaptability of fixed guidance strategy when working conditions change.

[0020] 2. This invention establishes a hierarchical system of operating conditions and a corresponding mapping relationship of guidance parameter sets. In simple operating conditions, lower processing parameters are used to save computing resources, while in complex operating conditions, higher processing parameters are used to ensure guidance accuracy. This achieves the optimal balance between computing resources and guidance accuracy, and the overall system response speed is improved compared with the prior art.

[0021] 3. The working condition classification model of this invention adopts a multi-feature fusion decision mechanism, which comprehensively considers three dimensions: environmental complexity, target object visibility, and interference level. The hierarchical judgment has a high accuracy rate, ensuring that the optimal guidance strategy can be selected in different industrial scenarios, significantly reducing the guidance failure rate and improving the success rate of robotic arm grasping. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall technical architecture of a vision-adaptive guidance method for robotic arms based on 3D point clouds. Figure 2 This is a schematic diagram of the core principle framework of the working condition sensing unit and adaptive parameter selection; Figure 3 This is a flowchart of the process framework for the operational feature analysis stage of point cloud data; Figure 4 This is a flowchart of the pose estimation and robot motion trajectory planning process based on adaptive parameters; Figure 5 This is a schematic diagram of the closed-loop control process of the dynamic update mechanism for operating condition levels. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 To be continued Figure 5 The present invention will be further described in detail below with reference to specific embodiments.

[0024] This invention provides a visual adaptive guidance method for a robotic arm based on 3D point clouds, including the following core steps: acquiring 3D point cloud data of the target scene, performing working condition feature analysis on the point cloud data, constructing a working condition classification model, adaptively selecting guidance parameters according to the working condition level, performing pose estimation based on the adaptive parameters, and planning the robotic arm's motion trajectory and performing grasping. The specific implementation methods of each step will be described in detail below.

[0025] The first step, S1, involves acquiring 3D point cloud data of the target scene. Step S1 serves as the data entry point for the entire vision-guided method, responsible for converting the physical spatial information of the target scene into digital point cloud data that can be processed by subsequent algorithms. This step encompasses sensor selection and ranging principle determination, frame rate setting to match the robot's movement rhythm, recording specifications for both spatial coordinates and reflection intensity, and defining the acquisition range and triggering synchronization control. Together, these constitute the complete conversion process of point cloud data from physical space to digital space, and are implemented according to the following steps.

[0026] Step S101: Select a 3D sensor and acquire depth information. After the robot arm vision guidance method is started, first determine whether to use a structured light sensor or a laser sensor as the data acquisition device.

[0027] A structured light sensor projects an coded grating pattern onto the surface of a target object, while a matching camera simultaneously captures the deformed light field image. The depth value of each pixel is calculated based on the principle of triangulation. Triangulation originates from the classic similar triangle theory in optical measurement. The specific calculation method is as follows:

[0028] in: This represents the depth of the target point along the optical axis of the sensor, in millimeters. The sensor baseline distance is the spatial distance between the optical center of the projector and the optical center of the camera, measured in millimeters. is the focal length of the camera lens, measured in millimeters; The parallax value represents the pixel offset of the same spatial point in the projected pattern and the acquired image, with the dimension in pixels. Physically, this formula means that the product of the baseline distance and the focal length, divided by the parallax, yields the vertical distance from the target point to the sensor plane. The spatial coordinates of each pixel are calculated by combining the depth value with the pixel coordinates and the camera intrinsic parameter matrix.

[0029] The laser sensor uses the time-of-flight method to measure distance. The time-of-flight method is based on the physics principle that distance traveled in uniform linear motion equals velocity multiplied by time. The time interval between the laser pulse emission and the received echo is recorded. Multiplying this interval by the speed of light and dividing by 2 gives the distance between the sensor and the target point. The specific formula is:

[0030] in: This represents the measured distance from the sensor to the target point, in meters. The speed of light, with a value of meters per second The time interval between the laser pulse emission time and the echo reception time is expressed in seconds.

[0031] Step S102: Set the acquisition frequency to match real-time requirements. The frame rate of the 3D sensor is set to 30 to 60 frames per second. The motion control cycle of the robotic arm is 20 to 50 milliseconds. The lower frame rate of 30 frames per second corresponds to a data update interval of approximately 33 milliseconds, ensuring the latest scene point cloud is obtained before each control cycle. The upper frame rate of 60 frames per second corresponds to a data update interval of approximately 17 milliseconds, suitable for high-speed operation scenarios. This frequency range can capture the pose changes of moving parts on the conveyor belt, providing a continuous and stable point cloud data stream for subsequent condition sensing units.

[0032] For applications involving high-speed sorting or dynamic tracking, the acquisition frequency is increased to over 60 frames per second. This reduces the data update interval to less than 16 milliseconds, allowing the temporal resolution of the point cloud data to match the robot's motion cycle.

[0033] Step S103 records two types of information: spatial coordinates and reflection intensity. Each frame of point cloud data carries two types of core information.

[0034] The first category is the three-dimensional spatial coordinates of the sampling points, including the horizontal coordinates. Vertical coordinates and depth coordinates This indicates the spatial position of each geometric sampling point on the surface of the target object in the sensor coordinate system. The value is obtained from the depth calculation formula in step S101. and From pixel coordinates and The values ​​are obtained through joint inverse calculation.

[0035] The second category is the reflection intensity value. The amplitude depends on the material type, color depth, and roughness of the target surface. Surfaces of the same material typically exhibit similar reflectance values, which can be used to distinguish between different material regions. At boundaries where material or color changes abruptly, reflectance intensity shows a step change, aiding in identifying the edge contours of the target object. The presence of an oxide layer, oil, or wear on the surface will reduce reflectance intensity, allowing for assessment of whether the surface condition affects grasping stability. Both types of information are stored together in the point cloud data structure for subsequent feature extraction and condition analysis modules to access.

[0036] Step S104: Define the acquisition range and trigger synchronous acquisition. The acquisition range needs to be preset, covering the entire workspace of the robot arm, including all areas where the target object may appear and the surrounding background environment. The three-dimensional boundaries are determined based on the kinematic reachable space of the robot arm: the lateral range covers the horizontal interval corresponding to the left and right swing amplitude of the robot arm base; the longitudinal range covers the front and rear interval corresponding to the extension length of the robot arm's upper arm; the vertical range covers the interval between the robot arm's lowest grasping height and its highest placement height.

[0037] During the acquisition process, the 3D sensor maintains a fixed pose, so that the point cloud data acquired at different times are aligned with a unified spatial reference.

[0038] When a target object enters the workspace and triggers a data acquisition signal, the sensor initiates a scan. The time window length for a single frame scan is consistent with the sensor's exposure cycle. At a frame rate of 30 frames per second, the window length is 33.3 milliseconds; at a frame rate of 60 frames per second, the window length is 16.7 milliseconds. The sensor completes a full frame scan within the window. At the end of the window, the point cloud data is transmitted to the preprocessing buffer, awaiting reading by the condition sensing unit.

[0039] In summary, step S1 transforms the three-dimensional spatial information of the target scene into a structured point cloud data stream, providing raw input for subsequent working condition feature analysis.

[0040] The next step is step S2, which involves analyzing the working condition features of the point cloud data. Step S2 introduces a working condition perception unit between point cloud acquisition and feature extraction. By analyzing the inherent attributes of the point cloud data in real time, the environmental complexity, target visibility, and interference level of the current working scenario are quantified, providing a basis for subsequent working condition classification and parameter adaptation. This is carried out according to the following steps.

[0041] Step S201: Set up the condition sensing unit and establish a parallel computing architecture. Deploy the condition sensing unit between the point cloud acquisition module and the feature extraction module. The condition sensing unit interfaces with the point cloud data stream in real time, completing the calculation of condition features before each frame of point cloud data enters the feature extraction stage. The condition sensing unit adopts a parallel computing architecture, using a graphics processor or a dedicated neural network acceleration chip as the computing platform, supporting a point cloud analysis throughput of more than 30 frames per second.

[0042] The specific deployment method of the parallel computing architecture is as follows: the point cloud data is divided into several data blocks according to spatial regions, each data block is assigned an independent computing thread, and each thread executes the corresponding feature calculation task in parallel. After the calculation is completed, the results of each data block are summarized and output. The parallel computing architecture ensures that the working condition perception unit can complete all feature extraction and quantization calculations within the acquisition interval of each frame of point cloud without affecting the overall real-time performance of the guidance method.

[0043] Step S202: Calculate the point cloud density distribution characteristics and use a voxel mesh generation method to quantitatively analyze the spatial distribution uniformity of the point cloud data. The collected point cloud data space is divided into a three-dimensional voxel mesh, where each voxel is a cubic unit with a fixed side length. The voxel side length is determined based on the typical size of the target object and the point cloud density, and is set to be within the range of 5 mm to 20 mm.

[0044] The number of point clouds falling into each voxel is counted, and the i-th... The number of point clouds for individual elements is The total number of voxels is Calculate the arithmetic mean of the number of voxel point clouds:

[0045] in This represents the arithmetic mean of the number of point clouds within each voxel.

[0046] Then calculate the standard deviation of the number of point clouds for each voxel:

[0047] in This indicates the number of point clouds within each voxel relative to the mean. The degree of dispersion.

[0048] The ratio of the standard deviation to the mean is defined as the evenness index of the density distribution, denoted as . :

[0049] This reflects the spatial clustering of point clouds in the target scene. When objects are uniformly distributed within the workspace, the difference in the number of point clouds per voxel is small, and the standard deviation is relatively small. relative mean Smaller A lower value indicates a lower complexity of the current working environment.

[0050] When objects are stacked, mutually occluded, or densely arranged, the point cloud within some voxels is highly concentrated while the point cloud within other voxels is sparse, and the standard deviation is [missing information]. relative mean Increase An increase in the value indicates a higher level of complexity in the current working environment.

[0051] Step S203: Calculate the point cloud noise characteristics and use the point distance statistical variance method to quantify the noise level of the point cloud data. For each sampling point in the point cloud... Retrieve its nearest neighbor in three-dimensional space Neighboring points, The value is set according to the point cloud density, and the range is 5 to 15. The sampling points are calculated separately. With The Euclidean distance between n neighboring points, the i-th The sampling point and its i.e. The Euclidean distance between 3 neighboring points is denoted as . For each sampling point, calculate its relationship with... Local variance of the Euclidean distance between neighboring points:

[0052] in Indicates the first Local distance variance of each sampling point; For the first The sampling point and its i.e. Euclidean distance between neighboring points; Indicates sampling point and The arithmetic mean of the Euclidean distances between n nearest neighbors is calculated using the following formula: .

[0053] Local variance over all sampling points The global average value is taken as the quantitative index of point cloud noise level, denoted as . :

[0054] in This represents the total number of sampling points in the point cloud.

[0055] The physical meaning is: when the point cloud data is relatively smooth and the sampling points are regularly distributed along the target surface, the variation in distance between each sampling point and its neighboring points is small. A lower value indicates stable illumination and good sensor accuracy. When the point cloud data contains many outliers or the surface sampling is uneven due to changes in illumination, the variation in distance between each sampling point and its neighbors increases. An increase in the value indicates that the noise interference is more severe under the current operating conditions.

[0056] Step S204: Calculate the surface normal consistency characteristics, and use the covariance matrix eigenvalue analysis method to quantitatively evaluate the smoothness and visibility of the target object's surface. For each sampling point in the point cloud... Select the one that is closest in three-dimensional space. A neighborhood set is formed by 10 neighboring points. The numerical values ​​range from 10 to 30. For the neighborhood set of the sampling points, the covariance matrix is ​​constructed as follows:

[0057] in Indicates the first The covariance matrix of the neighborhood set of each sampling point; Represents the th sampling point in the neighborhood set. Three-dimensional spatial coordinate vectors of neighboring points; Let be the centroid coordinate vector of the neighborhood set, i.e., the arithmetic mean of the spatial coordinates of all neighboring points; superscript This represents the matrix transpose operation.

[0058] covariance matrix The eigenvalue decomposition yields three eigenvalues, which are denoted as follows: (the eigenvalues ​​are then decomposed into three eigenvalues, arranged in descending order of magnitude). , , ,satisfy .

[0059] For each sampling point, calculate the ratio of its maximum eigenvalue to its minimum eigenvalue. and the ratio of the largest eigenvalue to the second largest eigenvalue To obtain a quantitative indicator that monotonically increases within the interval of 0 to 1, the reciprocals of the two ratios are taken respectively. and The arithmetic mean of the two reciprocal ratios of all sampled points in the point cloud is calculated, then summed and averaged. The average is then subtracted from 1 to obtain the global surface normal consistency index, denoted as [index missing]. :

[0060] in This represents the total number of sampling points in the point cloud.

[0061] When the surface where the sampling point is located is an ideally flat plane and Both are 0, and both ratios are 0. A value of 1 indicates good surface normal uniformity, and that the target surface is flat, intact, and visible. When the surface where the sampling point is located is an isotropic sphere or the point cloud is randomly diffusely distributed, the three characteristic values ​​are equal, and two ratios are both 1. A value of 0 indicates poor normal uniformity, an uneven target surface, or low visibility. Under actual working conditions... The value ranges from 0 to 1. A higher value indicates a smoother surface and better target visibility, while a lower value indicates a rougher surface, more severe occlusion, or poorer target visibility.

[0062] Step S205: Output multi-dimensional feature vector. The working condition sensing unit will use the density distribution features calculated in steps S202, S203, and S204. Noise characteristics and surface normal consistency features Combined into a three-dimensional feature vector The combination operation is a parallel arrangement of three independent scalar values, with no cross-coupling operations between the dimensions. , , Each component is calculated independently and output as one of the three components of the vector. The feature vector is output to the working condition classification model at the end of the current frame point cloud processing cycle, serving as input data for subsequent working condition level determination.

[0063] In summary, step S2 converts the raw point cloud data into a 3D feature vector. This feature vector numerically reflects the object density, signal-noise level, and target surface integrity of the current scene. Step S3 determines the working condition level based on this feature vector, thereby triggering the corresponding parameter selection mechanism.

[0064] The next step is S3, which involves constructing a working condition classification model. Step S3 receives the three-dimensional feature vector output from step S2 and classifies the current work scenario into different working condition levels through classification decisions. This enables subsequent steps to call the corresponding set of guiding parameters based on the level labels, including the following steps.

[0065] Step S301: Construct a three-dimensional feature vector, which incorporates the density distribution features output by the working condition sensing unit in step S2. Noise characteristics and surface normal consistency features Combined into a three-dimensional feature vector This serves as the input to the working condition classification model.

[0066] This is an index of density distribution uniformity; the lower the value, the more uniformly objects are distributed in the scene. This is a quantitative indicator of point cloud noise level; the lower the value, the smoother the point cloud. This is a surface normal consistency index; a higher value indicates a smoother target surface. The three feature quantities are calculated independently and then arranged in parallel, with no cross-coupling calculations between the dimensions.

[0067] Step S302: Set the threshold parameters for working condition classification, and use the threshold hierarchical decision method to determine the level of the current working condition. Set two sets of parameters: the first threshold and the second threshold.

[0068] because , and The physical meanings of the three features are different, and each of the three features corresponds to an independent first threshold and an independent second threshold, denoted as follows: , , as well as , , The first subscript indicates the corresponding feature number, and the second subscript 1 indicates the first threshold and 2 indicates the second threshold.

[0069] Each threshold was determined through statistical analysis of experimental data. Specifically, multiple frames of point cloud data were collected for three scenarios: simple, general, and complex operating conditions, and three feature values ​​were calculated for each frame. and These positive indicators statistically analyze the distribution range of characteristic values ​​under various working conditions, using the upper boundary of the characteristic value distribution under simple working conditions as the first threshold and the upper boundary of the characteristic value distribution under general working conditions as the second threshold.

[0070] for These inverse indices use the lower boundary of the characteristic value distribution under simple operating conditions as the first threshold and the lower boundary of the characteristic value distribution under general operating conditions as the second threshold. Once each threshold is determined, it remains fixed in subsequent applications and does not need to be recalculated.

[0071] and As a positive indicator, a higher value indicates a more complex working condition. Therefore, the first threshold is the upper limit, and the second threshold is higher than the first threshold. As a reverse indicator, a higher value indicates a simpler working condition. Therefore, the first threshold is the lower limit, and the second threshold is lower than the first threshold.

[0072] Step S303: Perform threshold comparison and working condition level determination, with the determination rules executed from high to low priority.

[0073] The first priority is to determine simple working conditions. When and and When the current working condition is determined to be a simple working condition, the following conditions are met: the target object surface is flat, there is no obstruction or interference from the surroundings, the lighting conditions are stable, and the point cloud data quality is high. Taking a single independent part placed on a conveyor belt under uniform lighting as an example, the simple working condition is determined when all three conditions are met simultaneously.

[0074] If the above conditions are not met, proceed to the second priority stage to determine the general operating condition. and and When the condition is normal, it is determined to be a normal working condition. The working scenario corresponding to normal working condition is: the surface of the target object has slight unevenness, there is a small amount of occlusion or slight changes in lighting conditions, and the point cloud data has a certain degree of noise.

[0075] If the above conditions are still not met, that is or or When this occurs, the current working condition is determined to be a complex working condition.

[0076] Complex working conditions correspond to scenarios such as: severely uneven target object surfaces, significant occlusion or object stacking, drastic changes in lighting conditions, and high noise levels in point cloud data. Taking a scenario with multiple stacked parts under direct sunlight as an example, the stacking leads to… The rise in elevation and direct sunlight introduce noise. Increased size, surface discontinuity The condition will decrease, and any condition exceeding the second threshold will trigger a complex working condition judgment.

[0077] In the above comparison process, the comparison operations of the three dimensions do not affect each other, and the final judgment result is a logical combination of the comparison results of the three dimensions.

[0078] Step S304: A machine learning classifier is used as an alternative to the threshold-based hierarchical decision method to classify operating conditions. Optional classification algorithms include support vector machines, random forests, or neural networks. The input to the classifier is the three-dimensional feature vector from step S301. The output is one of three level labels: simple working condition, general working condition, or complex working condition.

[0079] When a neural network is used as the classifier, the network structure is a three-layer fully connected feedforward network. The input layer receives a 3-dimensional feature vector. The hidden layer contains 64 nodes, and the activation function is ReLU (Rectified Linear Unit). The output layer contains 3 nodes, corresponding to the three categories, and the activation function is Softmax. This transforms the output value into a probability distribution of the three categories, and the category with the highest probability value is the working condition level label output by the classifier. The hidden layer and the output layer are fully connected.

[0080] The classifier's training set consists of point cloud data frames collected from multiple robotic arm operation sites at different time periods, under different lighting conditions, and with different object arrangements. Each sample is manually labeled as one of three levels based on simultaneously recorded scene images and objective measurements such as the number of objects, the number of stacked layers, the proportion of occlusion area, and illuminance.

[0081] The labeling rules are as follows: Simple working conditions correspond to single-layer placement, no occlusion, uniform lighting, and point cloud with no significant noise; General working conditions correspond to scenes with slight stacking or minor occlusion, localized lighting changes, or point cloud with minor noise; Complex working conditions correspond to scenes with multi-layer stacking, severe occlusion, drastic lighting changes, or point cloud with a large amount of noise. The classifier learns the mapping relationship between the 3D feature vector and the working condition level.

[0082] If a neural network is used as the classifier, the loss function is multi-class cross-entropy loss. The input is the predicted probability vector of the three classes output by the classifier, and the output is a measure of the difference between the predicted probability distribution and the labeled value. During training, the learning rate is set between 0.001 and 0.01, the batch size is set between 32 and 128, and the optimizer is an adaptive moment estimation optimizer. The training convergence objective is to achieve a classification accuracy of over 95% on the validation set. After training, the classifier can directly output the corresponding work condition level label based on the input feature vector.

[0083] Compared to threshold-based hierarchical decision-making methods, machine learning classifiers can handle non-linear relationships between feature values. When there is coupling or cross-influence between three feature values, the classifier can make a comprehensive judgment based on the distribution patterns in the training samples, without the need to manually set independent thresholds for each feature.

[0084] Step S305: Output the operating condition level label. The operating condition level label obtained in step S303 or step S304 is output to the adaptive selection module for the guidance strategy parameters. The operating condition level label is one of three: simple operating condition, general operating condition, or complex operating condition.

[0085] After the working condition level is determined, step S4 will select the corresponding set of guiding parameters from the mapping table based on the level label.

[0086] The next step is step S4 in the method of this embodiment. The guiding parameters are adaptively selected according to the working condition level. Step S4 receives the working condition level label output in step S3, selects a corresponding set of parameters from the parameter set according to the preset mapping relationship, and sends them to the feature extraction module and pose estimation module to complete the dynamic switching. The following steps are implemented.

[0087] Step S401: Establish a mapping table between working condition levels and guidance parameter sets. A mapping table containing the correspondence between three level labels and three sets of parameter sets is pre-established. The parameter sets contain four core parameters: point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration count, and convergence threshold.

[0088] The three level labels represent simple working conditions, general working conditions, and complex working conditions, with each level label uniquely corresponding to a set of parameter configurations. The mapping table is stored in a tabular format, with the level label serving as the index key and the corresponding four-dimensional parameter combination as the index value. When a lookup operation is performed, the level label is taken as input, and the output is the specific values ​​of all four parameters corresponding to that label.

[0089] Step S402: Configure the parameter sets corresponding to different working condition levels. The guidance parameter set configuration for the simple working condition is as follows: The point cloud preprocessing filter kernel size is 2 mm to 3 mm, used to remove a small number of noise points while retaining the detailed features of the target object. The feature extraction sampling rate is 80% to 100%. The pose estimation iteration count is 10 to 20 times. The convergence threshold is 0.01 m to 0.02 m.

[0090] The typical operating condition requires the following parameter set configuration: point cloud preprocessing filter kernel size is 4 mm to 6 mm; feature extraction sampling rate is 50% to 70%; pose estimation iterations are 30 to 50; and the convergence threshold is 0.005 m to 0.01 m.

[0091] The following parameter set configurations are used for complex operating conditions: point cloud preprocessing filter kernel size is 8 mm to 12 mm; feature extraction sampling rate is 30% to 50%; pose estimation iterations are 80 to 120; and convergence threshold is 0.001 m to 0.005 m.

[0092] The parameter sets at three levels form a gradient configuration. The point cloud preprocessing filter kernel size increases with the increase of operating complexity, the feature extraction sampling rate decreases with the increase of operating complexity, the pose estimation iteration number increases with the increase of operating complexity, and the convergence threshold decreases with the increase of operating complexity.

[0093] The parameter range is set based on the empirical matching relationship between point cloud quality and computing resources under various working conditions. Under simple working conditions, the point cloud quality is high, and accuracy can be guaranteed by using a smaller point cloud preprocessing filter kernel, a higher feature extraction sampling rate, fewer pose estimation iterations, and a more relaxed convergence threshold.

[0094] In complex operating conditions, noise and occlusion are severe, necessitating the increase of the point cloud preprocessing filter kernel to remove noise, the reduction of the feature extraction sampling rate to control computational load, the increase of pose estimation iterations, and the tightening of the convergence threshold to ensure convergence accuracy. The specific numerical ranges of each parameter were obtained through experimental verification in various typical operating scenarios.

[0095] The impact of different parameter combinations on pose estimation accuracy and computation time was tested in uniform lighting without occlusion, scenes with slight occlusion and lighting fluctuations, and scenes with severe stacking and strong noise. The selection principle was to achieve the required accuracy and the shortest computation time, and finally the above three parameter ranges were determined.

[0096] Step S403: Receive the operating condition level label and search for the corresponding parameter set. The adaptive selection module for the guidance strategy parameter receives the operating condition level label from step S303 or S304. Using the level label as an index, the adaptive selection module performs a search operation in the mapping table established in step S401, matching the received level label with the index keys in the mapping table one by one. Upon successful matching, it extracts the corresponding four parameter values: the point cloud preprocessing filter kernel size, the feature extraction sampling rate percentage, the integer value of the pose estimation iteration count, and the floating-point value of the convergence threshold. If the received level label cannot match any index key, the parameter set corresponding to the general operating condition is used as the alternative configuration by default.

[0097] In step S404, the parameter set is sent and dynamic switching is completed. The adaptive selection module for guiding strategy parameters sends the four found parameter values ​​to the feature extraction module and the pose estimation module. After receiving the point cloud preprocessing filter kernel size and feature extraction sampling rate, the feature extraction module uses the new parameters to perform operations in the point cloud preprocessing and feature extraction stages.

[0098] After receiving the pose estimation iteration count and convergence threshold, the pose estimation module performs calculations using the new parameters during the iterative optimization phase. Once both modules have completed parameter updates, the guidance strategy is dynamically switched.

[0099] After the parameter switching is completed, step S5 will perform feature extraction and pose estimation under the new parameters, thereby generating an adaptive response to different working conditions.

[0100] Then, in step S5 of this embodiment, pose estimation is performed based on adaptive parameters. The point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number and convergence threshold provided in step S4 are received. These parameters are used to perform calculations in the point cloud preprocessing, feature extraction and pose optimization stages to obtain the three-dimensional pose information of the target object, including the following steps.

[0101] Step S501: Perform point cloud preprocessing by filtering the original point cloud using the point cloud preprocessing filter kernel size selected in step S4. When the filter kernel size is 2 mm to 3 mm, voxel filtering or statistical filtering is performed to remove outlier noise while preserving the fine surface structure of the target object.

[0102] When the filter kernel size is 4 mm to 6 mm, it moderately smooths the point cloud surface and filters out moderate noise. When the filter kernel size is 8 mm to 12 mm, it forces smoothing of point clouds that are severely uneven or contain a lot of noise.

[0103] Step S502: Extract point cloud feature points. Feature points are extracted from the preprocessed point cloud data using the feature extraction sampling rate selected in step S4. Each feature point consists of two parts: keypoints and descriptors.

[0104] Keypoint detection employs any one of the ISS, Harris3D, or SIFT3D algorithms to identify the coordinate locations of significant curvature changes or local geometric structures in the point cloud. Descriptor computation uses any one of FPFH, SHOT, or ISS features to encode the geometric distribution of each keypoint's neighborhood into a numerical vector.

[0105] The sampling rate controls the number of feature points as follows: Keypoint detection is performed on the entire preprocessed point cloud to obtain a candidate keypoint set, and the salience score of each candidate keypoint is calculated. When the sampling rate is 80% to 100%, the number of keypoints corresponding to the top percentage of the sampling rate in terms of salience score is retained from the candidate keypoint set. When the sampling rate is 50% to 70%, the number of keypoints corresponding to the top percentage of the sampling rate is retained. When the sampling rate is 30% to 50%, the number of keypoints corresponding to the top percentage of the sampling rate is retained.

[0106] Step S503: Select a pose estimation algorithm, and use an improved iterative nearest point algorithm or a pose regression network based on deep learning for pose estimation.

[0107] The improved iterative nearest-point algorithm employs the G-ICP algorithm or the point-to-plane ICP algorithm, improving registration accuracy in noisy point clouds by optimizing the matching strategy and residual calculation method. The deep learning-based pose regression network uses PointNet++ or DGCNN networks to directly regress the 3D pose parameters of the target object from the feature point set.

[0108] The pose regression network used in step S5 is a pre-trained model. The training set construction method, loss function design and training convergence target in the training process have been disclosed in step S304. In this step, the trained network model is directly applied to perform pose regression.

[0109] Step S504: Perform iterative optimization and output pose results. Use the pose estimation algorithm selected in step S503 and perform pose optimization calculations based on the pose estimation iteration number and convergence threshold selected in step S4.

[0110] The objective function for iterative optimization is the sum of squared distances from each point in the transformed source point cloud to its nearest corresponding point in the target point cloud. This objective function is the standard form of the iterative nearest-point algorithm. In each iteration, the matching error under the current transformation matrix is ​​calculated, and the current matching error is compared with the previous matching error to calculate the absolute value of the error change.

[0111] When the absolute value is less than the convergence threshold, the optimization is considered converged and the iteration is terminated. When the number of iterations reaches the set maximum limit, the iteration is forcibly terminated. Under both termination conditions, the current transformation matrix is ​​used as the output.

[0112] When the number of iterations is 10 to 20, a small number of iterations are sufficient to reach convergence. When the number of iterations is 30 to 50, a moderate number of iterations is used to handle moderate noise and occlusion. When the number of iterations is 80 to 120, a large number of iterations are used to find the pose transformation matrix that minimizes the matching error under conditions of severe noise and occlusion.

[0113] A relaxed termination condition is used when the convergence threshold is 0.01 m to 0.02 m, a moderate termination condition is used when the convergence threshold is 0.005 m to 0.01 m, and a strict termination condition is used when the convergence threshold is 0.001 m to 0.005 m.

[0114] After the iteration terminates, the three-dimensional pose information of the target object is output. The pose information includes two parts: position coordinates and attitude angles. The position coordinates are the X-axis, Y-axis, and Z-axis coordinates in the sensor coordinate system or the robot's base coordinate system. The attitude angles are the Euler angles α, β, and γ of the rotations around the X, Y, and Z axes, or output as quaternions qw, qx, qy, and qz. The pose result is passed to step S6 for robot motion trajectory planning.

[0115] Taking the automotive parts sorting scenario as an example, after step S3 determines that the current working condition is complex, step S4 corresponds to the output point cloud preprocessing with a filter kernel size of 10 mm, a feature extraction sampling rate of 35%, a pose estimation iteration count of 100 times, and a convergence threshold of 0.002 meters.

[0116] Step S501 uses a 10mm filter to filter the point cloud of the stacked parts, removing outlier noise caused by direct sunlight. Step S502 extracts the top 35% of feature points with saliency scores at a 35% sampling rate. Step S504 performs iterative nearest-point optimization with an iteration limit of 100 iterations and a convergence threshold of 0.002 meters. The iteration terminates when the error change drops to 0.0015 meters (less than 0.002 meters) after the 78th iteration, and the pose transformation matrix is ​​output. The positional deviation between this matrix and the manually measured actual pose is 0.003 meters, and the angular deviation is 0.8 degrees.

[0117] Finally, in step S6, the robot arm's motion trajectory is planned and the grasping is executed. Step S6 receives the target object's three-dimensional pose information output in step S5. Through four stages—coordinate system transformation, path planning, grasping execution, and dynamic updating—the robot arm is controlled to complete the grasping and handling operation of the target object, including the following steps.

[0118] Step S601: Transform the pose information from the sensor coordinate system to the robot's base coordinate system, and use a hand-eye calibration method to obtain the extrinsic parameter relationship between the 3D sensor and the robot. The extrinsic parameter relationship includes the rotation matrix and translation vector.

[0119] Multiply the position coordinates of the target object in the sensor coordinate system output in step S5 by the rotation matrix on the left, and then add the translation vector to obtain the position coordinates of the target object in the robot's base coordinate system. Multiply the rotation matrix formed by the attitude angles of the target object in the sensor coordinate system with the rotation matrix in the extrinsic parameters to obtain the attitude angles of the target object in the robot's base coordinate system.

[0120] Step S602: Generate motion trajectory. Based on the target object pose transformed in step S601, and combined with the manipulator kinematic model, a trajectory planning algorithm is used to generate a motion path from the current position of the manipulator end effector to the target object grasping position.

[0121] During execution, the trajectory planning algorithm reads the angle limit value, speed limit value, and acceleration limit value of each joint of the robot, as well as the torque output limit value of each joint drive motor, and uses the above constraints as the boundary conditions for path search.

[0122] The trajectory planning algorithm can employ any one of the following: fast expanding random tree algorithm, probabilistic road map algorithm, or multinomial interpolation-based algorithm. Taking the fast expanding random tree algorithm as an example, the fast expanding random tree algorithm randomly samples nodes in the robot's configuration space, taking the current position as the starting point and the target grasping position as the ending point, and gradually expands the path tree until the starting point and the ending point are connected, generating a continuous path that satisfies joint angle constraints and velocity constraints.

[0123] The sampling step size of the fast expanding random tree algorithm is set to 10 mm to 50 mm depending on the size of the robot's workspace, and the maximum number of iterations is set to 5000 to 20000. The output is the motion path of the robot's end effector from its current position to the target grasping position, represented as a sequence of temporal trajectory points in joint space.

[0124] Step S603: Perform the grasping action. After the robot arm moves to the pre-grabbing position near the target object according to the motion trajectory generated in step S602, adjust the posture of the robot arm's end effector so that the orientation of the end effector matches the target object's posture angle after the conversion in step S601.

[0125] The grasping pose is determined based on the shape and center of gravity distribution of the target object, using the surface area near the target object's center of mass or geometric center as the grasping point. Based on the target object's position coordinates and attitude angles obtained in step S601, and the target object's geometric dimensions obtained in the feature extraction stage of step S5, the position of the target object's center of mass is calculated. Using the target object's center of mass as the position coordinate of the grasping point, the gripping center of the end effector coincides with the grasping point, while the orientation of the end effector is aligned with the normal direction of the target object's surface at the grasping point.

[0126] The grasping action can be performed using any of the following methods: suction, clamping, or wrapping, depending on the shape and material of the target object.

[0127] Step S604: Transport the object to the unloading position and release it. After gripping, the robot arm carries the target object from the current gripping position to the unloading position along a predetermined transport path. The end point of the transport path is the unloading position, and the coordinates of the unloading position are pre-calibrated in the robot arm's base coordinate system. After the robot arm reaches the unloading position, the end effector releases the target object, completing the gripping and transporting operation of a single work cycle.

[0128] Step S605: Perform dynamic working condition monitoring and parameter switching. During the process of the robot arm performing steps S602 to S604, continuously collect point cloud data of the target scene at a frequency of 5 to 10 frames per second. The collection frequency of 5 to 10 frames per second is lower than the point cloud collection frequency in step S1 and is only used for working condition monitoring.

[0129] The collected point cloud data is input into the working condition feature analysis module in step S2 and the working condition level determination module in step S3 to perform working condition feature extraction and working condition level determination. The specific implementation methods of the working condition feature analysis module and the working condition level determination module have been disclosed in steps S2 and S3 respectively. In step S605, the above modules are directly called to perform working condition monitoring.

[0130] When a change in operating condition level is detected, a parameter set switching process is triggered. The parameter set switching adopts a smooth transition method: starting from the moment the level change is detected, the parameter values ​​of the current parameter set are gradually transitioned to the parameter values ​​corresponding to the new level using a linear interpolation method.

[0131] The linear interpolation calculation method is as follows: Determine the total transition time T, divide the difference between the current parameter value and the target parameter value by the number of control cycles within the total transition time T to obtain the adjustment step size for each control cycle, and increase or decrease the current parameter value by one adjustment step size in each control cycle until the target parameter value is reached. The total transition time T is set within the range of 0.5 seconds to 2 seconds.

[0132] After the switch is completed, steps S5 and S6 continue to perform subsequent point cloud feature extraction, pose estimation and trajectory planning operations using the new parameter set until the next change in working condition level is detected.

[0133] To facilitate understanding, the specific application process of the method in this embodiment will be explained using an industrial parts sorting scenario as an example.

[0134] On an automated production line at an auto parts manufacturing company, robotic arms sort various metal parts on a conveyor belt. The parts are diverse in type and shape, and may stack or tilt during transport. Lighting conditions in the production workshop vary over time; direct afternoon sunlight causes noise in the point cloud data collected by sensors.

[0135] After startup, the 3D sensor acquires 3D point cloud data of the conveyor belt area at a frequency of 30 frames per second. Initially, the parts on the conveyor belt are sparsely distributed, and the surfaces of individual parts are flat. At this time, the feature values ​​obtained by the condition perception unit are: density distribution feature F1 is 0.12, noise feature F2 is 0.08, and surface normal consistency feature F3 is 0.15. All three feature values ​​are below their respective first thresholds, and the condition classification model determines it to be a simple condition. The adaptive selection module for the guidance strategy parameters selects a simple condition parameter set: point cloud preprocessing filter kernel size of 2 mm, feature extraction sampling rate of 100%, pose estimation iterations of 15, and convergence threshold of 0.01 m. The pose estimation module calculates the part pose under these parameters, and the robotic arm completes the grasping and sorting operations.

[0136] After running for a period of time, the number of parts on the conveyor belt increased, resulting in multiple parts stacking. Direct afternoon sunlight caused noise in some point cloud data. The feature values ​​obtained from the condition perception unit analysis became: F1 = 0.65, F2 = 0.55, and F3 = 0.58. Both F1 and F2 exceeded their respective second thresholds, and the condition classification model determined it to be a complex condition. The adaptive selection module for the guidance strategy parameters switched the parameter set to the complex condition configuration: point cloud preprocessing filter kernel size of 10 mm, feature extraction sampling rate of 35%, pose estimation iterations of 100, and convergence threshold of 0.002 m. Although the computation time increased, the pose estimation accuracy was guaranteed, and the robotic arm could still accurately grasp the stacked parts.

[0137] Once the part is moved to the shaded area, the operating condition characteristic values ​​return to the normal operating condition range. The adaptive selection module for the guidance strategy parameters automatically switches to the normal operating condition parameter set, improving processing speed while maintaining a certain level of accuracy. Throughout the entire process, this embodiment dynamically adjusts the guidance parameters according to changes in operating conditions to achieve adaptive guidance.

[0138] On the other hand, the 3D point cloud-based robotic arm vision adaptive guidance system disclosed in this invention includes: The point cloud acquisition module is used to acquire 3D point cloud data of the target scene. The working condition perception unit is deployed between the point cloud acquisition module and the feature extraction module. The working condition perception unit receives each frame of point cloud data output by the point cloud acquisition module. The working condition perception unit is used to calculate the density distribution uniformity index, point cloud noise level quantization index and surface normal consistency index for each frame of point cloud data, and combine them into a three-dimensional feature vector. The working condition classification model module receives the three-dimensional feature vector output by the working condition sensing unit and is used to determine the current working condition level based on the three-dimensional feature vector. The adaptive selection module for guidance strategy parameters receives the working condition level label output by the working condition classification model module. The adaptive selection module for guidance strategy parameters is used to select the corresponding set of guidance parameters from the preset mapping relationship according to the working condition level. The set of guidance parameters includes at least the point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number and convergence threshold. The feature extraction module receives the point cloud preprocessing filter kernel size and feature extraction sampling rate output by the guiding strategy parameter adaptive selection module. The feature extraction module is used to perform point cloud preprocessing and feature extraction using the point cloud preprocessing filter kernel size and feature extraction sampling rate. The pose estimation module receives the pose estimation iteration number and convergence threshold output by the guidance strategy parameter adaptive selection module. The pose estimation module is used to perform pose estimation with the pose estimation iteration number and convergence threshold, and outputs the three-dimensional pose information of the target object. The trajectory planning and execution module receives the 3D pose information output by the pose estimation module. The trajectory planning and execution module is used to plan the robot's motion trajectory based on the 3D pose information and execute the grasping operation.

[0139] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0140] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A vision-adaptive guidance method for a robotic arm based on 3D point clouds, comprising the steps of acquiring 3D point cloud data of a target scene, and the steps of feature extraction and pose estimation of the 3D point cloud data of the target scene; characterized in that Also includes: Between the point cloud acquisition step and the feature extraction step, the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index are calculated for each frame of point cloud data and combined into a three-dimensional feature vector. The current operating condition level is determined based on the three-dimensional feature vector. Based on the operating condition level, a corresponding set of guiding parameters is selected from a preset mapping relationship. The set of guiding parameters includes at least the point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number, and convergence threshold. Point cloud preprocessing, feature extraction, and pose estimation are performed using the selected set of guiding parameters.

2. The method of claim 1, wherein: The density distribution uniformity index is calculated as follows: The point cloud data space is divided into a three-dimensional voxel grid, and the number of point clouds of each voxel is counted , the total number of voxels is , and the total number of voxels is ; Computing the arithmetic mean of all voxel point cloud quantities ; Calculate the standard deviation of the number of point clouds for each voxel. ; The ratio of standard deviation to mean is defined as the density distribution uniformity index. .

3. The method according to claim 1, characterized in that: The point cloud noise level quantification index is calculated as follows: For each sampling point in the point cloud Retrieve the sampling points The closest in three-dimensional space 1 neighboring point; Calculate the sampling points With the Local variance of Euclidean distance between neighboring points ,in For the first The sampling point and the first The sampling point of the first sampling point Euclidean distance between neighboring points For the sampling point With the The arithmetic mean of the Euclidean distances between the nearest neighbors. ; Local variance over all sampling points The global average value is used as the quantitative indicator of point cloud noise level. , This represents the total number of sampling points in the point cloud.

4. The method according to claim 1, characterized in that: The surface normal consistency index is calculated as follows: For each sampling point in the point cloud Select the sampling points The closest in three-dimensional space A neighborhood set is formed by a number of neighboring points; Constructing the covariance matrix ,in Indicates the sampling point The first in the neighborhood set Three-dimensional spatial coordinate vectors of neighboring points Let be the centroid coordinate vector of the neighborhood set; For the covariance matrix Eigenvalue decomposition yields three eigenvalues. , , Sort in descending order; Calculate the global surface normal consistency index , This represents the total number of sampling points in the point cloud.

5. The method according to claim 1, characterized in that: The current operating condition level is determined using a threshold-based hierarchical decision-making method. Three feature quantities are defined, each with a corresponding first and second threshold. These three feature quantities are density distribution uniformity indices. Point cloud noise level quantification index and surface normal consistency index ; When density distribution uniformity index Less than or equal to the density distribution uniformity index The corresponding first threshold And point cloud noise level quantification index Less than or equal to the point cloud noise level quantization index The corresponding first threshold And the surface normal consistency index Greater than or equal to the surface normal consistency index The corresponding first threshold At that time, it was determined to be a simple working condition; When density distribution uniformity index Less than or equal to the density distribution uniformity index The corresponding second threshold And point cloud noise level quantification index Less than or equal to the point cloud noise level quantization index The corresponding second threshold And the surface normal consistency index Greater than or equal to the surface normal consistency index The corresponding second threshold At that time, it was determined to be a normal working condition; When density distribution uniformity index Greater than the density distribution uniformity index The corresponding second threshold Or point cloud noise level quantification index Greater than the point cloud noise level quantization index The corresponding second threshold Or surface normal consistency index Smaller than the surface normal consistency index The corresponding second threshold At that time, it was determined to be a complex working condition.

6. The method according to claim 1, characterized in that: The determination of the current operating condition level uses a machine learning classifier; The machine learning classifier is a three-layer fully connected feedforward network. The input layer of the three-layer fully connected feedforward network receives the three-dimensional feature vector, and the hidden layers of the three-layer fully connected feedforward network contain... The output layer of the three-layer fully connected feedforward network contains nodes and employs a linear rectified unit activation function. The output layer outputs three level labels—simple working condition label, general working condition label, and complex working condition label—using the Softmax activation function. The category with the highest probability value is used as the working condition level label output by the machine learning classifier.

7. The method according to claim 1, characterized in that: The set of guiding parameters: The guiding parameter set corresponding to the simple working condition is: the point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice; The typical operating condition's corresponding guidance parameter set is: Point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice; The guidance parameter set corresponding to complex working conditions is: the point cloud preprocessing filter kernel size is... millimeters to millimeters, feature extraction sampling rate to The pose estimation iteration number is Next The convergence threshold is... Rice to rice.

8. The method according to claim 1, characterized in that: During the process of generating motion trajectories, performing gripping actions, transporting materials to the unloading position, and releasing them, the robotic arm operates at a rate of [per second]. Frame to The frame rate is used to continuously collect point cloud data of the target scene, and to extract working condition features and determine the working condition level. When a change in operating condition level is detected, a guided parameter set switching process is triggered. This process uses linear interpolation to gradually transition the parameter values ​​of the current parameter set to the parameter values ​​corresponding to the new level. The total transition time is set to [missing information]. Instant Second.

9. The method according to claim 1, characterized in that: The steps of calculating the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index are performed in the working condition sensing unit. The working condition perception unit adopts a parallel computing architecture, which is as follows: the point cloud data is divided into several data blocks according to spatial regions, each data block is assigned an independent computing thread, each thread executes the corresponding feature calculation task in parallel, and the results of each data block are summarized and output after the calculation is completed.

10. A 3D point cloud-based robotic arm vision adaptive guidance system, applied to the method described in any one of claims 1-9, characterized in that: include: A point cloud acquisition module, which is used to acquire three-dimensional point cloud data of the target scene; The working condition perception unit is deployed between the point cloud acquisition module and the feature extraction module. The working condition perception unit receives each frame of point cloud data output by the point cloud acquisition module. The working condition perception unit is used to calculate the density distribution uniformity index, the point cloud noise level quantization index, and the surface normal consistency index for each frame of point cloud data, and combine them into a three-dimensional feature vector. The working condition classification model module receives the three-dimensional feature vector output by the working condition sensing unit, and is used to determine the current working condition level based on the three-dimensional feature vector. The adaptive selection module for guiding strategy parameters receives the working condition level label output by the working condition classification model module. The adaptive selection module for guiding strategy parameters is used to select the corresponding set of guiding parameters from a preset mapping relationship based on the working condition level. The set of guiding parameters includes at least the point cloud preprocessing filter kernel size, feature extraction sampling rate, pose estimation iteration number and convergence threshold. The feature extraction module receives the point cloud preprocessing filter kernel size and feature extraction sampling rate output by the guidance strategy parameter adaptive selection module. The feature extraction module is used to perform point cloud preprocessing and feature extraction using the point cloud preprocessing filter kernel size and the feature extraction sampling rate. The pose estimation module receives the pose estimation iteration number and convergence threshold output by the guidance strategy parameter adaptive selection module. The pose estimation module is used to perform pose estimation using the pose estimation iteration number and the convergence threshold, and outputs the three-dimensional pose information of the target object. The trajectory planning and execution module receives the three-dimensional pose information output by the pose estimation module, and is used to plan the motion trajectory of the robot arm and perform the grasping operation based on the three-dimensional pose information.

Images