A point cloud data rapid processing system based on 3D vision

Abstract

The application discloses a kind of point cloud data fast processing systems based on 3D vision, it is related to point cloud data processing technical field.Binocular acquisition module generates high-precision point cloud of anti-interference;Pretreatment module reduces noise and dimensionality, retains key geometric features;Layered feature extraction module fuses curvature weight and non-maximum suppression strategy, improves the robustness of ORB algorithm in rotation, scale and noise scene;Industrial scene segmentation module accurately separates stacked objects through normal vector clustering and region growing algorithm, and combines lightweight semantic model secondary verification boundary;Two-stage registration module dynamically adjusts threshold based on error feedback, realizes pose optimization from coarse registration to fine registration;Coordinate mapping module establishes space mapping model through multi-pose calibration and robot multi-axis linkage trajectory optimization, solve the problem that 3D machine vision algorithm exists in prior art higher latency, improve the real-time and rapidity of 3D model recognition and feature extraction, adapt to the efficiency requirement of industrial production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud data processing, and particularly relates to a point cloud data rapid processing system based on 3D vision. BACKGROUND

[0002] With the evolution of industrial automation towards intelligence, robot guidance technology based on 3D machine vision has become a core demand in the field of intelligent manufacturing. However, the existing three-dimensional vision system generally faces the bottleneck problems of high algorithm processing delay, poor adaptability to dynamic scenes, and insufficient precision of multi-coordinate system conversion in industrial scenes, which seriously restricts the improvement of industrial production efficiency. Especially in high-speed pipeline and precision assembly scenes, traditional technologies are difficult to balance real-time and detection accuracy, resulting in a decrease in robot grasping success rate and system response lag, which becomes a key obstacle to the upgrading of industrial automation.

[0003] At the algorithm level, traditional three-dimensional point cloud processing technology has significant efficiency defects. Existing methods mostly use iterative closest point (ICP) registration, SIFT feature extraction and other computationally intensive algorithms, which can guarantee a certain accuracy, but when facing high-density point cloud data in industrial scenes, the computational complexity increases exponentially, resulting in long time consumption of feature extraction and model reconstruction. In addition, point cloud dimension reduction relies on random sampling consensus (RANSAC) and other probability models, which is difficult to achieve stable dimension reduction while preserving key features, and often causes loss of object edge information due to oversimplification, directly affecting the accuracy of subsequent segmentation and registration.

[0004] At the system integration level, the multi-coordinate system conversion system of the existing three-dimensional vision system has the problem of error accumulation. The checkerboard calibration method relied on by camera calibration in industrial scenes is easily affected by mechanical vibration and temperature drift, resulting in misalignment of internal and external parameters, which causes deviation in the mapping relationship from image pixel coordinates to robot base coordinates. At the same time, the existing system lacks a real-time coordinate compensation mechanism for dynamic targets, and when the workpiece position is slightly displaced due to conveyor shaking, the system is difficult to correct the three-dimensional space coordinates in time, ultimately causing the robot grasping path to deviate from the actual target position.

[0005] The complex environment of industrial scenes further amplifies the defects of existing technologies. In the detection of strong light reflection metal surfaces, semi-transparent packaging materials or densely stacked objects, traditional structured light projection is prone to mirror reflection, light scattering and other interference, resulting in missing or distortion of point cloud data. Existing algorithms often alleviate noise by increasing the number of filtering, but this exacerbates the processing delay, forming a vicious cycle of "precision-efficiency". The existing system is forced to compromise between reducing detection accuracy (increasing the rate of missed detection) and sacrificing production efficiency (slowing down the pace), making it difficult to meet the stringent requirements of "zero defects, zero delay" in the era of Industry 4.0. SUMMARY

[0006] In view of the above-mentioned defects of the prior art, the application provides a point cloud data rapid processing system based on 3D vision.

[0007] To achieve the above object, the application is implemented by the following technical solutions:

[0008] A point cloud data rapid processing system based on 3D vision comprises:

[0009] A binocular acquisition module is used to acquire multi-view images of a target object and generate high-precision three-dimensional point cloud data.

[0010] A point cloud preprocessing module is used to perform noise filtering and dimension reduction processing on the original point cloud data, and specifically comprises:

[0011] The noise filtering is based on neighborhood statistical filtering and voxel grid filtering, removes outliers and performs uniform sampling.

[0012] The dimension reduction processing is performed by using a principal component analysis (PCA) algorithm to reduce dimensions and retain main direction features.

[0013] A hierarchical feature extraction module is used to perform feature extraction on the preprocessed point cloud data by using an improved ORB algorithm, generate a point cloud feature descriptor, and specifically comprises:

[0014] A multi-scale Gaussian pyramid is constructed, and a scale-invariant feature detection space is generated by layer-by-layer down-sampling.

[0015] Curvature weight and non-maximum suppression are fused to perform feature screening and retain feature points.

[0016] Redundant feature points are removed through spatial distribution optimization.

[0017] An industrial scene segmentation module is based on normal vector clustering and region growing algorithm, synchronously separates stacked object point clouds, and performs feedback correction on the point cloud feature descriptor to generate corrected point cloud data.

[0018] A two-stage registration module sequentially performs a coarse registration stage and a fine registration stage on the corrected point cloud data to generate a three-dimensional reconstruction model.

[0019] A coordinate mapping module establishes a multi-coordinate system mapping relationship based on the three-dimensional reconstruction model, outputs a six-degree-of-freedom pose instruction, and performs real-time transmission.

[0020] Advantages

[0021] Compared with the known prior art, the technical solutions provided by the application have the following advantages:

[0022] 1. The application significantly improves the efficiency, accuracy and scene adaptability of three-dimensional point cloud processing through multi-module collaborative optimization and dynamic adaptive mechanism. Based on the acquisition module of multi-frequency phase encoding and binocular vision fusion, it can effectively resist environmental light interference and generate high-precision and high-robustness three-dimensional point cloud data, providing a reliable data basis for object reconstruction and pose calculation in complex industrial scenes; the preprocessing module combines neighborhood statistical filtering and adaptive voxel grid dimension reduction technology to remove noise while retaining key geometric features, solving the contradiction between data redundancy and feature loss in traditional methods, and providing high-quality input for subsequent processing steps. The hierarchical feature extraction module significantly enhances the robustness of the ORB algorithm to rotation, scale change and surface details through curvature weight optimization and non-maximum suppression strategy, so that the feature matching accuracy is improved to more than 95%, meeting the real-time demand of industry.

[0023] 2. Through the collaborative design of hierarchical processing architecture and dynamic parameter adjustment unit, the real-time performance and environmental adaptability of the system are considered. Through multi-level cooperative processing mechanism, combined with neighborhood statistical filtering, adaptive voxel grid filtering and principal component analysis (PCA) algorithm, the problems of data redundancy, noise interference and insufficient feature retention in traditional methods are solved. The coordinate mapping module innovatively integrates multi-pose calibration and kinematic parameter optimization algorithm, and establishes a hand-eye calibration system including distortion compensation model, realizing sub-millimeter level pose mapping error control from image space to robot operation space. The creativity of the technical scheme also lies in the deep coupling of the safety protection unit and the processing flow. The real-time calculation of the joint interference potential of the robot ensures the operation safety under complex working conditions when the collision risk is detected.

[0024] 3. The application effectively solves the common problems of traditional three-dimensional vision system in complex scenes, such as accuracy decay, insufficient real-time performance and poor environmental adaptability. Through the hierarchical feature pyramid structure and multi-scale optimization strategy, the system still maintains stable feature extraction ability under harsh working conditions such as strong light and high occlusion. The dynamic resource scheduling unit intelligently allocates computing resources according to the point cloud processing load. The "collection-analysis-control" closed-loop technology chain innovatively constructed by the scheme deeply integrates three-dimensional vision perception and robot motion control, providing a high-precision and high-reliability vision guidance solution for the field of intelligent manufacturing. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0026] Figure 1 This is a system module connection diagram of the present invention;

[0027] Figure 2 This is a diagram of the vision system platform of the present invention;

[0028] Figure 3 This is a flowchart illustrating the algorithm improvement of the present invention;

[0029] Figure 4 This is a flowchart of the 3D vision system of the present invention;

[0030] Figure 5 This is a coordinate relationship diagram of the hand-eye system of the present invention;

[0031] Figure 6 This is a module connection diagram of the dynamic parameter adjustment unit 160 and the safety protection unit 170 of the present invention. Detailed Implementation

[0032] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0033] Application Overview:

[0034] In existing technologies, point cloud processing systems based on 3D vision often rely on the independent optimization of a single algorithm module, making it difficult to achieve efficient end-to-end parsing in complex industrial scenarios. Traditional methods often suffer from registration errors due to dimensionality reduction distortion and insufficient feature robustness when processing high-noise, large-scale point cloud data. This is especially problematic in dynamic operating scenarios, where existing systems lack the ability to adaptively adjust to changes in point cloud density, resulting in processing delays and accuracy degradation. Existing equipment generally uses feature extraction and registration models with fixed parameters, failing to dynamically optimize the processing flow based on workpiece surface characteristics. This leads to persistently high parsing failure rates under complex conditions such as weak texture and strong reflection, making it difficult to meet the stringent real-time and reliability requirements of intelligent manufacturing.

[0035] To solve the above problems, the inventors found that the collaborative optimization of point cloud feature stability and dimension reduction strategy is the key to improve system performance. Through experimental verification, principal component analysis (PCA) extracts the main features of the data in the field of pattern recognition, image processing, etc., and improves the performance and generalization ability of the model. Further research found that the discriminability of traditional ORB features in three-dimensional point clouds is significantly affected by the surface wrinkle density. By introducing a local curvature weight and a non-maximum suppression strategy, the robustness of feature matching can be effectively improved. Based on the above findings, the inventors constructed a dynamic parameter adjustment unit, which feeds back the point cloud density, curvature distribution and processing error to each module in real time, forming a closed-loop optimization system.

[0036] The system first generates high-precision point cloud data through a binocular vision module, and a preprocessing module simultaneously performs statistical filtering and voxel grid filtering downsampling. The curvature distribution analysis selects the PCA algorithm for dimension reduction. The feature extraction module constructs a multi-scale ORB feature pyramid and fuses a local curvature weight factor in the traditional BRIEF descriptor to optimize the spatial distribution. The scene segmentation module uses a hybrid strategy of spectral clustering vector pre-segmentation and dynamic region growing to accurately extract the workpiece boundary. The two-stage registration module constructs a probability correspondence graph based on the improved ORB feature in the coarse registration stage, and implements ICP optimization with normal vector constraint in the fine registration stage, combined with KD-Tree accelerated search and dynamic distance threshold shrinkage mechanism. The coordinate mapping module establishes a sub-millimeter level pose mapping model through multi-axis linkage calibration, and the dynamic adjustment unit monitors the processing error in real time and feeds back key parameters such as dimension reduction rate, feature dimension, etc., forming a closed-loop control.

[0037] Compared with the prior art, the traditional point cloud processing system mostly uses fixed dimension reduction algorithm and static feature extraction parameters, which is easy to cause feature loss and registration failure in complex curved surface workpiece detection. The present scheme innovatively constructs a curvature-driven PCA processing architecture, which significantly improves the analytical reliability of the system in weak texture and high noise conditions through three technical breakthroughs of dynamic dimension reduction mode switching, curvature weight feature optimization and closed-loop parameter adjustment. Compared with the prior art, the feature matching accuracy and registration efficiency of the present scheme are significantly improved under the same hardware conditions, and the error accumulation in dynamic operation can be effectively suppressed. The multi-module collaborative optimization design realizes the balance between processing accuracy and real-time performance, and provides a high-robustness three-dimensional vision solution for intelligent manufacturing.

[0038] After introducing the basic concept of the present application, the embodiments of the present application will be specifically introduced with reference to the accompanying drawings.

[0039] Embodiment

[0040] A point cloud data rapid processing system based on 3D vision of the present embodiment, as shown in Figure 1 , includes:

[0041] The binocular acquisition module 100 acquires multi-view images of the target object and generates high-precision three-dimensional point cloud data.

[0042] As shown in the figure, the embodiment adopts a Mech-mind binocular 3D camera, which includes two cameras and a depth sensor. The 3D camera takes pictures, the light machine projects structured light onto the photographed object, the camera acquires images, and the acquired images are operated to generate point clouds through Ethernet communication with the computer. Figure 2

[0043] In the embodiment, the binocular acquisition module 100 of the Mech-mind binocular 3D camera plays an important role. The two cameras and a depth sensor contained therein work cooperatively, the two cameras acquire images of the target object from different angles, and they are precisely synchronized to capture images of the target scene at the same time. By comparing and analyzing the images captured by the two cameras, the depth information of the objects in the scene can be obtained, and more comprehensive visual information of the target object can be obtained through multi-view acquisition. The depth sensor is responsible for measuring the distance information of each point of the target object to the camera, and the image information acquired by the camera is combined.

[0044] When acquiring images, the light emitting device of the 3D camera will project structured light onto the photographed object, and the camera can capture the object image with the structured light pattern, etc. By decoding the captured image with coding information, the depth value of each point in the scene can be calculated, and the depth image resistant to environmental light interference can be generated through multi-frequency phase coding and converted into high-precision three-dimensional point cloud data. In the embodiment, triple-frequency phase coding (1200Hz / 1400Hz / 1600Hz) is adopted, the absolute phase value is calculated through the least square phase unwrapping algorithm, and then the high-precision three-dimensional point cloud data acquired is transmitted to the connected computer through the built-in communication module in the form of Ethernet.

[0045] Due to the adoption of multi-frequency phase coding technology, the system can better resist the interference of external environmental light, object surface reflection characteristics and other factors, thereby generating more accurate and reliable depth maps. The depth image is a kind of image, in which the value of each pixel represents the distance between the scene point corresponding to the pixel and the sensor, and it can directly show the three-dimensional structure information of the scene.

[0046] ​Compared with the traditional technology, the traditional measurement method often relies on a single sensor or simple image acquisition technology, and has defects such as incomplete data information, easy to be disturbed by the environment, low measurement accuracy, etc. For example, when acquiring three-dimensional information of an object, the traditional technology can only obtain two-dimensional images of part of the perspective, and it is difficult to comprehensively and accurately reflect the real three-dimensional structure of the object, and when facing complex environment or change of object surface characteristics, the stability and accuracy of the measurement result will be greatly reduced. The dual acquisition module 100 of the Mech-mind dual 3D camera used in the scheme can effectively solve the technical problems of single data, insufficient precision and poor anti-interference ability in the traditional technology through multi-view image acquisition and cooperation of the depth sensor, and can obtain more accurate three-dimensional information of the object.

[0047] Through the above technical scheme, the present application realizes comprehensive and high-precision three-dimensional information acquisition of the target object. The dual acquisition module of the dual 3D camera works cooperatively to capture images of the target object from different perspectives and combine the distance information measured by the depth sensor, and through multi-frequency phase encoding and other technical processing, it effectively resists the interference of external environmental light and object surface reflection characteristics and the like, generates accurate and reliable depth images and high-precision three-dimensional point cloud data. These data are transmitted to the connected computer in the form of Ethernet through the built-in communication module, providing a rich and accurate three-dimensional data basis for subsequent related analysis and processing, so that the entire system can stably and accurately acquire the three-dimensional structure information of the target object in a complex environment, and the modules closely cooperate and efficiently interact, ensuring the comprehensiveness, accuracy and stability of the three-dimensional information acquisition.

[0048] The point cloud preprocessing module 110: carries out noise filtering and dimension reduction processing on the original three-dimensional point cloud data, including:

[0049] The noise filtering is based on neighborhood statistical filtering and voxel grid filtering, removes outliers and performs uniform sampling;

[0050] The dimension reduction processing adopts principal component analysis (PCA) algorithm for dimension reduction, and retains the main direction features.

[0051] The neighborhood statistical filtering dynamically removes noise points exceeding a preset threshold by calculating the standard deviation of the neighborhood distance of the point cloud; the voxel grid filtering uniformly down-samples the point cloud at an adaptive resolution.

[0052] The original three-dimensional point cloud data obtained by the 3D camera contains hundreds of thousands or even millions of points, and a large number of points are redundant for expressing effective information of an object and will bring extremely high operation cost to subsequent point cloud processing algorithms. Therefore, the point cloud data must be dimensionally reduced and sampled to make the point cloud more concise. PCA (Principal Component Analysis) is a commonly used dimension reduction processing algorithm at present, which is a linear dimension reduction method and can retain rich three-dimensional information without distortion. The central idea is that a point in a high-dimensional coordinate space is multiplied by a matrix to be mapped to a low-dimensional coordinate space, and the matrix is a diagonalized covariance matrix, which aims to eliminate redundant data.

[0053] As shown in Figure 3 The present application mainly improves the algorithm for real-time detection and rapid processing, and the overall idea is to first use the PCA (Principal Component Analysis) algorithm to reduce the dimension of the original point cloud collected, then use the ORB feature extraction algorithm to quickly extract the key points, then perform object segmentation on the extracted point cloud, and finally obtain a three-dimensional model through point cloud registration.

[0054] The point cloud preprocessing module 110 mainly performs a series of preprocessing operations on the original point cloud data. The purpose is to optimize the data and provide better input for subsequent feature extraction and other operations. Remove noise points in the original point cloud data introduced by factors such as collection device precision and environmental interference. These noise points may interfere with subsequent feature extraction and analysis, leading to inaccurate results. At the same time, perform data normalization to make point cloud data from different sources or under different collection conditions have certain consistency in numerical range, etc., for unified processing.

[0055] When the original point cloud data is obtained, the point cloud preprocessing module 110 first uses neighborhood statistical filtering to remove outliers. The principle is to calculate the standard deviation of the neighborhood distance of the point cloud, and dynamically remove noise points that deviate significantly from the normal data distribution according to a pre-set threshold. For example, if the average distance of a point to its surrounding neighborhood points exceeds the set threshold too much, it will be judged as an outlier and removed.

[0056] The standard deviation calculation formula of neighborhood statistical filtering is as follows:

[0057]

[0058] Where σ is the neighborhood distance standard deviation, which measures the degree of data dispersion;

[0059] is the Euclidean distance from the i-th point in the neighborhood to the center point;

[0060] : the average distance of all points in the neighborhood;

[0061] N: number of points in the neighborhood.

[0062] Example: When the average distance of a point neighborhood is 10 mm, and the standard deviation is 2 mm, if the preset threshold is 3σ, then the points with a distance greater than μ+3σ=16 mm are determined as outliers and removed. = 2mm, if the preset threshold is 3σ, then the points with a distance greater than μ+3σ=16 mm are determined as outliers and removed.

[0063] At the same time, voxel grid filtering also plays an important role in this process. Voxel grid filtering uniformly down-samples the original point cloud with adaptive resolution. Specifically, it divides the three-dimensional space into small voxel grid units, and determines the retained points in each unit by certain rules (such as taking the center point or average value, etc.), so as to reduce the amount of point cloud data while better preserving its overall spatial features, so that the computational complexity of the point cloud in subsequent processing is reduced.

[0064] The amount of point cloud data obtained by the original 3D camera can be extremely large, containing hundreds of thousands or even millions of points. However, there is a lot of redundant data in terms of effective information of the object. In order to make the subsequent point cloud processing algorithm run more efficiently, the point cloud must be processed by dimensionality reduction. PCA is a linear dimensionality reduction method, and its core idea is to map a point in a high-dimensional coordinate space to a low-dimensional coordinate space by multiplying it with a specific matrix (i.e. the diagonalized covariance matrix). Covariance matrix and eigenvalue decomposition are used. In this way, redundant data can be greatly eliminated while preserving rich three-dimensional information. For example, in a three-dimensional point cloud data, if the data in a certain direction changes relatively small, i.e. the information in that direction is relatively redundant, the PCA algorithm can map it to a low-dimensional space through the corresponding matrix transformation, so that the point cloud is more concise, providing a more efficient data basis for subsequent point cloud processing such as feature extraction, model reconstruction, etc.

[0065] The formula for PCA algorithm covariance matrix and eigenvalue decomposition is as follows:

[0066]

[0067] where C: covariance matrix of point cloud data;

[0068] x i : three-dimensional coordinates of the i-th point in the point cloud;

[0069] μ: mean vector of point cloud data, .

[0070] ​The eigenvectors corresponding to the first k largest eigenvalues of the covariance matrix C are selected as the principal directions. For example, if the first two principal components are retained (k = 2), the dimension of the projected point cloud is reduced from 3D to 2D.

[0071] Through the preprocessing operation of the point cloud preprocessing module 110, the original three-dimensional point cloud data is effectively filtered and dimensionally reduced, laying a good foundation for the subsequent steps of the entire three-dimensional data processing process.

[0072] Examples:

[0073] Experimental configuration:

[0074] Test platform: Intel Xeon Gold 6230R + NVIDIA Quadro RTX 8000

[0075] Dataset: ModelNet40 industrial component dataset (including 20 typical parts such as gears and bearings).

[0076] Experimental data:

[0077]

[0078] Compared with the traditional technology, the traditional point cloud processing method usually relies on a single filtering algorithm or a fixed parameter dimension reduction strategy, and has defects such as high data redundancy, strong noise sensitivity, and poor scene adaptability. For example, the traditional neighborhood filtering uses a static threshold to remove outliers, which is difficult to cope with the dynamic point cloud density distribution in industrial scenes; the dimension reduction algorithm such as random sampling easily loses key geometric features, resulting in insufficient accuracy of subsequent segmentation and registration. The present scheme solves the problems of data redundancy, noise interference and insufficient feature retention in traditional methods through a multi-level cooperative processing mechanism, combining neighborhood statistical filtering, adaptive voxel grid filtering and principal component analysis (PCA) algorithm, significantly improving the point cloud processing efficiency and industrial scene adaptability.

[0079] Through the above technical solutions, this application achieves efficient preprocessing and accurate feature extraction of complex industrial point cloud data. The point cloud preprocessing module effectively eliminates environmental interference and retains key geometric information of the object surface through dynamic noise filtering and adaptive dimensionality reduction, providing high-quality data input for subsequent feature extraction and segmentation. The hierarchical feature extraction module combines an improved ORB algorithm and multi-scale spatial optimization to enhance robustness to rotation, scale changes, and surface details. The industrial scene segmentation module accurately separates the boundaries of stacked object point clouds through the synergistic effect of normal vector clustering and dynamic region growing. All modules interact closely based on a unified data flow, reducing computational complexity while ensuring the accuracy of 3D reconstruction and robot grasping pose calculation, providing a highly reliable and real-time point cloud processing solution for industrial automation scenarios.

[0080] Hierarchical feature extraction module 120: used to extract features from the preprocessed point cloud data using an improved ORB algorithm to generate point cloud feature descriptors, specifically including:

[0081] Construct a multi-scale Gaussian pyramid and generate a scale-invariant feature detection space by downsampling layer by layer;

[0082] Feature selection is performed by fusing curvature weighting and nonmaximum suppression;

[0083] Redundant feature points are eliminated through spatial distribution optimization.

[0084] The point cloud feature descriptor is generated and its dimensions are dynamically adjusted to adapt to the scene complexity.

[0085] In the feature selection process of fusing curvature weights and non-maximum suppression, non-maximum suppression prioritizes retaining edge points and feature points with significant curvature changes in key point detection, and removes outliers through curvature consistency verification.

[0086] like Figure 4 The diagram shows the overall operation flow of the 3D vision system, including the point cloud preprocessing module 110 and the hierarchical feature extraction module 120. The purpose of keypoint extraction is to reduce computation by calculating feature vectors for only a subset of points and then finding corresponding point pairs based on feature vector matching. ORB feature extraction is based on FAST feature point detection and description technology, possessing scale and rotation invariance, as well as invariance to noise and perspective affine transformations.

[0087] Gaussian pyramid is a multi-scale image representation method widely used in image processing and computer vision. In this module, a series of images with different resolutions are obtained by successive Gaussian smoothing and downsampling operations on the original image. These images with different resolutions constitute the multi-scale Gaussian pyramid. The multi-scale Gaussian pyramid performs layer-by-layer Gaussian smoothing and downsampling on the original point cloud to generate a scale-invariant feature detection space. The significance lies in that different scales of images can capture different levels of feature information. Large-scale images can capture macro features such as the overall outline of an object, while small-scale images can capture detailed features such as the surface texture of an object. When extracting features from a 3D point cloud of a mechanical part, large-scale features can quickly locate the approximate shape of the part, and small-scale features can accurately extract some subtle texture features on the surface of the part. Through Gaussian pyramid and principal direction correction, ORB features can still be matched stably when the object rotates (such as changes in robot grasping posture) and the scale changes (such as switching between close-up and wide-angle views).

[0088] Curvature weight refers to assigning weights based on the local curvature of the point cloud (such as the degree of concave and convex of the surface). The feature points in areas with significant curvature changes (such as part edges and grooves) have higher priority. Non-maximum suppression refers to retaining only the feature points with the largest response value in the neighborhood and suppressing redundant points. The feature point response value is calculated as: R = a FAST score + β curvature value, where a and β are weight coefficients.

[0089] In the feature screening process, integrating curvature weight and non-maximum suppression is a key operation. The non-maximum suppression prioritizes edge points and feature points with significant curvature changes in key point detection and removes abnormal points through curvature consistency verification. The introduction of curvature weight takes into account the importance of different point curvatures in point cloud data for features. Areas with large curvature changes often contain more valuable feature information. By giving these points higher curvature weights, important feature points can be more effectively retained in the feature screening process. Non-maximum suppression compares the response values of feature points within a certain neighborhood and suppresses non-maximum points to select more representative feature points. In a point cloud data of an object with complex surfaces, points with large curvature changes at edges and corners will be preferentially retained as feature points due to the combined effect of curvature weight and non-maximum suppression, while points in flat areas may be suppressed because they contain less feature information.

[0090] In the feature extraction process, some redundant feature points can be generated, which can increase the subsequent calculation amount and can interfere with the final analysis result. Through the way of spatial distribution optimization, according to the distribution of feature points in space, redundant feature points that are too dense or have little contribution to the overall feature representation can be removed. For example, in a relatively uniform plane area, if there are a large number of dense feature points, spatial distribution optimization can appropriately reduce the number of these points and only keep a small number of representative points, thereby reducing the data amount and computational complexity without losing key feature information.

[0091] After the above operation, the point cloud feature descriptor is finally generated, and its dimension is dynamically adjusted according to the complexity of the specific application scenario. In a relatively simple scenario, a lower-dimensional feature descriptor can be used to reduce the calculation amount and storage requirement; while in a complex scenario, a higher-dimensional feature descriptor can be needed to more comprehensively and accurately represent the features of the point cloud to meet the needs of subsequent analysis, recognition and other operations.

[0092] In the current mainstream feature extraction algorithm, there are SIFT scale invariant feature transform algorithm, SURF accelerated robust feature algorithm and ORB fast feature extraction algorithm, etc. Among them, the SIFT algorithm has the advantages of scale invariance, but the computational complexity is relatively high; the SURF algorithm improves the calculation speed to some extent, but the stability and accuracy of the features can be affected in some cases. The ORB algorithm is used for feature extraction in the present application mainly based on the actual needs of industrial production. Industrial production requires real-time feature detection, and the speed of the algorithm is a key factor to be considered comprehensively. ORB feature extraction is based on FAST feature detection and description technology, which has scale and rotation invariance, and also has certain invariance to noise and perspective affine. This enables it to quickly complete feature extraction operation under the premise of ensuring a certain feature extraction accuracy in the industrial 3D vision system, meets the real-time requirements, and is effectively applied to part detection on automatic production lines, robot vision navigation and other scenes that require fast and accurate acquisition of 3D point cloud features.

[0093] Examples:

[0094] Experimental method: In the ABB IRB 4600 robot work unit, collect bolt point clouds (surface reflectance level ISO 8501-1: Sa2.5) under different poses. The test conditions are light intensity 5000±200 lux, and point cloud density 4000 points / cm³.

[0095] Data analysis:

[0096]

[0097] Compared with the traditional technology, the traditional point cloud feature extraction method usually relies on single scale feature detection or fixed weight screening strategy, and has defects such as high feature redundancy, poor rotation and scale adaptability, and low calculation efficiency. For example, although the SIFT algorithm has scale invariance, its high calculation complexity is difficult to meet the real-time demand of industry; although the SURF algorithm improves the speed, the feature stability is easily disturbed by noise, and the key geometric details are easily lost on complex curved surface. The scheme improves the ORB algorithm, combines multi-scale Gaussian pyramid, curvature weight fusion and non-maximum suppression, solves the problems of feature redundancy, noise sensitivity and insufficient scene adaptability in traditional methods, and significantly improves the robustness and efficiency of feature extraction.

[0098] Through the above technical solutions, the present application realizes efficient extraction and accurate description of industrial point cloud features. The hierarchical feature extraction module captures the cooperative features of macroscopic contour and microscopic details through the multi-scale Gaussian pyramid, optimizes the curvature weight and screens out key points with high distinguishability through non-maximum suppression, effectively avoiding the interference of redundant features in flat areas; the spatial distribution optimization further eliminates dense redundant points, and dynamically adjusts the feature descriptor dimension to adapt to different scene complexity. Each module closely interacts based on preprocessed high-quality point cloud data, ensures rotation and scale invariance, significantly improves the feature matching accuracy and real-time performance in industrial scenes such as robot vision navigation and part detection, and provides reliable technical support for three-dimensional reconstruction and automation control in complex environments.

[0099] The industrial scene segmentation module 130: based on the normal vector clustering and region growing algorithm, the stacked object point cloud is separated synchronously, and the point cloud feature descriptor is feedback corrected to generate corrected point cloud data;

[0100] The normal vector clustering is based on the local normal vector distribution of the point cloud to construct spatial clustering, and the spectral clustering algorithm is used to divide the adjacent points with consistent normal vector direction into initial sub-regions for pre-segmentation; the region growing algorithm takes the initial sub-region as the growing seed, and iteratively expands the region boundary combined with the following conditions:

[0101] The dynamic threshold of the consistency of the normal vector direction of adjacent points is self-adaptively adjusted according to the local surface curvature change rate;

[0102] The deviation between the region growing direction and the principal curvature direction is not more than a preset angle threshold.

[0103] In the industrial scene segmentation module 130, the core is to separate the stacked object point cloud based on the normal vector clustering and the region growing algorithm. First, regarding the normal vector clustering part, it is based on the local normal vector distribution of the point cloud to construct spatial clustering. Specifically, it is realized by a specific spectral clustering algorithm, which can divide adjacent points into initial sub-regions according to the consistency of the normal vector direction of the point cloud, so as to perform pre-segmentation. The normal vector here refers to the vector perpendicular to a point on the surface of the point cloud, which plays an important role in describing the geometric characteristics of the point cloud.

[0104] The region growing algorithm takes the initial sub-regions obtained by the normal vector clustering as the growing seeds. In the region growing process, the following key conditions are combined to iteratively expand the region boundary:

[0105] 1. Adjacent point normal vector direction consistency dynamic threshold: In actual industrial scenes, the surface of the object may have certain irregularities, so a fixed threshold cannot be used to determine whether adjacent points belong to the same region. Therefore, the dynamic threshold of the adjacent point normal vector direction consistency needs to be adjusted adaptively according to the local surface curvature change rate. For example, in the region where the curvature changes relatively gently, the threshold can be appropriately relaxed; while in the region where the curvature changes sharply, the threshold needs to be tightened to ensure the accuracy of region division.

[0106] 2. The deviation between the region growing direction and the principal curvature direction does not exceed the preset angle threshold: The principal curvature direction reflects the directions of maximum and minimum curvature of the object surface at a point. When region growing, the deviation between the growing direction and the principal curvature direction is controlled within the preset angle threshold, which can make the segmented region more consistent with the actual geometric shape of the object, avoiding unreasonable region division.

[0107] Through the synchronous cooperation of the normal vector clustering and the region growing algorithm, the industrial scene segmentation module 130 can effectively and accurately segment the point cloud of the stacked object, thereby providing accurate point cloud data basis for the subsequent industrial processing process.

[0108] The industrial scene segmentation module 130 further introduces a feature descriptor feedback correction mechanism based on the separation of the stacked object point cloud by the normal vector clustering and region growing algorithm, dynamically optimizes the segmentation result, and generates corrected point cloud data. After completing the initial segmentation, the module realizes closed-loop optimization through the following process: first, based on the multi-scale feature descriptors generated by the hierarchical feature extraction module 120, the point cloud features of the segmentation boundary region are locally re-evaluated to identify potential mis-segmentation regions (such as feature blur regions caused by noise or surface reflection); then, the geometric and semantic features of the disputed region are jointly analyzed by combining a lightweight semantic segmentation model, focusing on key point cloud information through an attention mechanism, correcting the weight distribution of the feature descriptor, and suppressing abnormal feature interference; finally, according to the corrected feature descriptor, the segmentation boundary is recalculated to generate corrected point cloud data with higher geometric consistency and more accurate semantic information. This feedback correction mechanism can adapt to the surface characteristics of different industrial scenes (such as metal reflection, plastic weak texture, etc.) in real time, improving the segmentation accuracy by more than 15% in complex stacked scenes, and providing more robust data input for subsequent registration and coordinate mapping modules, significantly enhancing the system's analytical reliability in dynamic environments.

[0109] Compared with traditional technologies, traditional industrial point cloud segmentation methods rely on a single algorithm (such as Euclidean clustering or fixed threshold region growing), which has defects such as blurred boundaries of stacked objects, poor adaptability to surface geometry changes, etc. For example, Euclidean clustering cannot distinguish objects with similar normal vector directions but actual separation; fixed threshold region growing is prone to over-segmentation or under-segmentation in complex curved surface scenes. The present scheme solves the problem of insufficient segmentation accuracy caused by complex geometric features and rigid threshold setting in traditional methods through the cooperative mechanism of normal vector clustering pre-segmentation and dynamic region growing, significantly improving the accuracy and scene adaptability of stacked object point cloud separation.

[0110] Through the above technical solutions, the present application realizes accurate segmentation and efficient processing of industrial stacked object point cloud. The normal vector clustering module quickly extracts initial sub-regions with consistent normal vector directions through the spectral clustering algorithm, providing a geometric consistency basis for subsequent segmentation; the region growing algorithm combines adaptive threshold and principal curvature direction constraint to iteratively expand the region boundary, ensuring the continuity and reasonableness of the segmentation boundary in complex curved surface scenes. The two modules interact closely based on the geometric characteristics of the point cloud, effectively suppressing noise interference and mis-segmentation risk while preserving the actual shape of the object, providing high-reliability point cloud data support for robot grasping, assembly detection, and other industrial processes, significantly improving the operation accuracy and scene adaptability of the automation system.

[0111] The industrial scene segmentation module 130 performs conflict detection on overlapping regions, screens the optimal segmentation boundary based on curvature continuity and spatial connectivity, and performs secondary verification on controversial regions through a lightweight semantic segmentation model; the lightweight semantic segmentation model is a point cloud segmentation network based on an attention mechanism, which is adapted to different industrial scene object categories through transfer learning.

[0112] In this application, one of the core tasks of the industrial scene segmentation module 130 is to perform conflict detection on overlapping regions. In industrial scenes, due to various complex factors, there are often overlapping situations of different objects or regions, which requires the industrial scene segmentation module 130 to accurately handle through specific methods.

[0113] First, it screens the optimal segmentation boundary based on curvature continuity and spatial connectivity. Curvature continuity refers to the fact that the curvature change at the surface of an object or the boundary of a region should be continuous and smooth. By analyzing this characteristic, the possible segmentation boundary can be initially determined. Spatial connectivity is considered from the perspective of spatial distribution to ensure that the segmented regions have reasonable connectivity in space and avoid unreasonable segmentation results. Through the comprehensive use of these two characteristics, the relatively optimal segmentation boundary can be screened.

[0114] However, the segmentation boundary determined by the above method may still have some uncertainty or controversy. Therefore, the industrial scene segmentation module 130 will also perform secondary verification on controversial regions through a lightweight semantic segmentation model. The lightweight semantic segmentation model here is a point cloud segmentation network based on an attention mechanism. The attention mechanism plays a key role in focusing on key information in the model, allowing the model to pay more attention to point cloud features and other information that have a significant impact on the segmentation result, thereby improving the accuracy of segmentation.

[0115] Moreover, the lightweight semantic segmentation model adapts to different industrial scene object categories through transfer learning. Transfer learning is a machine learning technique that can use knowledge learned in one field (such as a typical industrial scene) such as object features, segmentation patterns, etc. to quickly apply to another related but possibly different field (other different industrial scenes), so that the model can handle object segmentation tasks in different industrial scenes without starting from scratch, greatly improving the model's versatility and efficiency.

[0116] Through the processing steps of the industrial scene segmentation module 130, the segmentation task in the industrial scene can be more accurately and efficiently completed, providing a reliable foundation for subsequent related industrial operations, etc.

[0117] Example:

[0118] Test conditions: 5 stacked gear point clouds (30000-50000 points per gear), occlusion rate 40%-60%.

[0119]

[0120] Compared with the traditional technology, the traditional industrial point cloud segmentation method relies on single geometric feature analysis or fixed model verification strategy, and has defects such as segmentation ambiguity of complex overlapping area, poor cross-scene adaptability, etc. For example, single rule based on curvature or connectivity is easy to cause boundary misjudgment due to local noise; static semantic model needs to be repeatedly trained for different scenes, which is time-consuming and has insufficient generalization ability. The scheme solves the problems of low segmentation accuracy of overlapping area, high model migration cost and weak adaptability to complex scenes in traditional methods through the synergistic mechanism of curvature continuity screening, spatial connectivity constraint and lightweight semantic segmentation model, and significantly improves the reliability of segmentation results and the universality of industrial scenes.

[0121] Through the above technical solutions, the present application realizes high-precision segmentation of industrial overlapping area point cloud and efficient adaptation to cross-scene. The curvature continuity analysis combined with spatial connectivity constraint quickly screens the optimal segmentation boundary and suppresses geometric noise interference; the lightweight semantic segmentation model focuses on the key features of the controversial area through the attention mechanism, and performs secondary verification on the preliminary segmentation result to ensure the boundary continuity and semantic rationality; the transfer learning technology further migrates the segmentation knowledge learned by the model in the typical scene to the new scene, reducing the data labeling and training cost. Based on the deep synergy of point cloud geometry and semantic characteristics, the modules significantly improve the segmentation accuracy and algorithm generalization ability in complex stacking, occlusion and multi-object interaction scenes, provide robust segmentation data support for robot grasping, assembly detection and other industrial operations, and promote the precise decision and efficient execution of automatic systems in dynamic environment.

[0122] The two-stage registration module 140: for the modified point cloud data, sequentially performing a coarse registration stage and a fine registration stage, dynamically adjusting the search strategy and convergence condition of the unknown pose point cloud through registration error feedback, generating a three-dimensional reconstruction model;

[0123] The coarse registration stage is based on ORB feature matching and RANSAC algorithm to screen matching point pairs, and calculates the initial rotation matrix and translation vector; the fine registration stage adopts an improved ICP algorithm, combines KD-Tree to accelerate nearest neighbor search, and optimizes through dynamic distance threshold;

[0124] The distance threshold optimization includes:

[0125] According to the weighted relationship between the maximum point distance of the current point cloud frame and the registration error residual, the distance threshold for corresponding point screening is dynamically adjusted;

[0126] In the KD-Tree accelerated search process, the normal vector consistency constraint is introduced, and only the nearest neighbor point pair with an included angle of the normal vector less than a set distance threshold is reserved;

[0127] Through an iterative error feedback mechanism, the distance threshold range is gradually contracted to approximate the optimal registration accuracy.

[0128] The purpose of registration is to solve the rotation matrix and translation vector of the source point cloud to the template point cloud. In the industrial production disorder scene, parts are stacked with each other and have different poses, so as to speed up the registration speed and accuracy. The application adopts two stages of coarse registration and fine registration. Coarse registration is a coarse matching of two unknown pose point cloud objects, and fine registration is to minimize the spatial pose difference between the two point clouds on the basis of coarse registration. The two-stage registration module 140 dynamically adjusts the search strategy and convergence condition of the unknown point cloud through registration error feedback.

[0129] Firstly, the coarse registration stage is based on ORB feature matching and RANSAC algorithm to screen matching point pairs, and then calculate the initial rotation matrix and translation vector. ORB (Oriented FAST and Rotated BRIEF) feature is a feature descriptor with rotation invariance and certain robustness to noise. It combines the advantages of FAST key point detection and BRIEF descriptor, and can quickly and effectively extract representative feature points in image or point cloud data. And the RANSAC (Random Sample Consensus) algorithm is an iterative method for estimating the parameters of a mathematical model from a data set containing a large amount of noise and outliers. In the coarse registration stage, it can screen out reliable matching point pairs that meet certain geometric consistency from a large number of possible matching point pairs, thereby providing accurate basis for subsequent calculation of initial rotation matrix and translation vector.

[0130] The steps of the ORB algorithm are as follows: use the FAST algorithm to detect feature points in the image. The FAST algorithm quickly judges whether a point is a feature point by comparing the brightness of the pixel point with its neighborhood pixels; non-maximum suppression is performed on the detected feature points to remove adjacent duplicate feature points, and only the feature points with the largest response value are retained; the main direction of each feature point is calculated to make the feature descriptor have rotation invariance; and the rBRIEF algorithm is used to generate a binary descriptor for each feature point. The rBRIEF descriptor is a binary string used to represent the local features of the feature point.

[0131] The improved ICP algorithm is used in the fine registration stage, combined with KD Tree to accelerate the nearest neighbor search, and the dynamic distance threshold optimization is used to further improve the registration accuracy. The ICP (Iterative Closest Point) algorithm is a classic point cloud registration algorithm, and its basic idea is to find the corresponding point pairs between two point clouds by continuously iterating, and to calculate the rigid body transformation (rotation and translation) that minimizes the sum of squared distances between the corresponding point pairs, so as to accurately align the two point clouds.

[0132] Improved ICP registration algorithm: objective function and dynamic distance threshold formula:

[0133]

[0134] Parameter meaning: R: rotation matrix;

[0135] t: translation vector;

[0136] p i : point in source point cloud;

[0137] q i : corresponding point in target point cloud;

[0138] w i : dynamic weight, controlled by distance threshold d max

[0139]

[0140] Adjusting the distance threshold based on the weighted relationship between the maximum point distance and the registration error residual is to dynamically adjust the distance threshold for corresponding point selection according to the weighted relationship between the maximum point distance and the registration error residual of the current point cloud frame. In the actual point cloud registration process, the distribution of point cloud and the error of registration are dynamically changing. Through the dynamic adjustment of the weighted relationship, the setting of the distance threshold can be more reasonable and adaptive, so as to more accurately select the effective corresponding point pairs and improve the accuracy and stability of registration.

[0141] ​KD Tree (K-Dimensional Tree) is a data structure used for efficient data retrieval in k-dimensional space. In this technology, KD Tree is used to quickly search for the nearest neighbor of each point in the point cloud data, greatly improving the efficiency of nearest neighbor search and accelerating the entire fine registration process. During the KD Tree accelerated search process, a normal vector consistency constraint is introduced, and only the nearest neighbor pairs with an included angle between normal vectors less than a set distance threshold are retained. This is because the normal vector of a point in the point cloud data contains its local geometric information. By introducing the normal vector consistency constraint, the nearest neighbor pairs that are more consistent with the geometric consistency can be further filtered out, avoiding the introduction of incorrect corresponding points due to pure distance measurement, thereby improving the accuracy of registration.

[0142] Through an iterative error feedback mechanism, the distance threshold range is gradually narrowed to approach the optimal registration accuracy. During the iterative process of registration, as the iteration progresses, the registration error gradually decreases, and accordingly, the range of the distance threshold can be gradually reduced. This allows the subsequent search and registration to focus more on the more accurate area and point pair, ultimately achieving high-precision point cloud registration.

[0143] Examples:

[0144] Experimental setup: The API Radian laser tracker (accuracy ± 0.5 μm) was used to measure the actual pose of the robot end effector.

[0145] Registration error:

[0146]

[0147] The core purpose of registration is to solve the rotation matrix and translation vector of the source point cloud to the template point cloud. In the industrial production disorder scene, due to the complex situation of parts stacking, position varying, etc., in order to speed up the registration speed and improve the registration accuracy, this project adopts two stages of coarse registration and fine registration. Coarse registration, as a preliminary matching of two unknown point cloud objects, can quickly determine the approximate relative position and attitude relationship, while fine registration is based on coarse registration, further minimizing the spatial pose difference between the two point clouds through more refined algorithms and optimization strategies, thereby achieving accurate point cloud registration and providing accurate point cloud data basis for subsequent industrial production related operations (such as part identification, assembly, etc.).

[0148] Compared with the traditional technology, the traditional point cloud registration method relies on single-stage algorithm (such as classic ICP) or fixed parameter matching strategy, and has defects such as low registration efficiency, error accumulation sensitivity, poor adaptability to complex scenes, etc. For example, although the ICP algorithm can achieve high-precision registration, it relies on initial pose assumption and has high computational complexity, and it is difficult to cope with problems such as part stacking and variable pose in industrial disorder scenes; the fixed distance threshold is easy to introduce error corresponding points under dynamic point cloud distribution, resulting in registration convergence failure. The scheme solves the technical problems such as difficult to balance registration speed and accuracy, poor robustness in complex scenes in traditional methods through the cooperative optimization mechanism of coarse registration and fine registration, and combines dynamic threshold adjustment and normal vector consistency constraint, which significantly improves the point cloud registration efficiency and reliability in industrial automation scenes.

[0149] Through the above technical scheme, the present application realizes efficient and accurate registration of industrial disorder point cloud. In the coarse registration stage, the ORB feature matching and RANSAC algorithm are used to quickly screen geometric consistency matching point pairs, providing a robust initial pose estimation for fine registration; in the fine registration stage, an improved ICP algorithm is used, combined with KD-Tree accelerated search and dynamic distance threshold optimization, to gradually reduce registration error and suppress abnormal point interference. The two-stage registration module dynamically adjusts the search strategy and convergence condition through the error feedback mechanism, while ensuring the real-time performance of the algorithm, and significantly improves the registration accuracy in complex stacking scenes. The modules cooperate closely based on a unified data stream, and gradually optimize the pose parameters from coarse to fine, providing a highly reliable spatial alignment data basis for industrial operations such as robot grasping and part assembly, and promoting the accurate decision-making and efficient execution of automated systems in dynamic environments.

[0150] The two-stage registration module 140 is aimed at the modified point cloud data output by the industrial scene segmentation module 130, and realizes the generation of a high-precision three-dimensional reconstruction model through the cooperative optimization of coarse registration and fine registration. Through the error feedback loop, the module adjusts the search strategy (such as local area search density) and convergence condition (such as the upper limit of iteration number) in real time, while ensuring real-time performance, and compresses the time consumption of fine registration to within 50ms, with a pose error of the reconstructed model and the real object less than 0.1mm.

[0151] In addition, the semantic information of the modified point cloud (such as the object category labeled by the lightweight semantic model) is integrated into the registration weight calculation, and the registration strategy is adaptively adjusted for different types of workpieces (such as gears and bolts). Through the above closed-loop optimization mechanism, the two-stage registration module achieves a registration success rate of more than 95% in complex industrial scenes, providing a three-dimensional reconstruction model with millimeter-level precision for robot grasping and assembly, and significantly improving the operation reliability and efficiency of the automated production line.

[0152] The coordinate mapping module 150: establishes a multi-coordinate system mapping relationship based on the three-dimensional reconstruction model, outputs a six-degree-of-freedom pose instruction, and transmits it in real time.

[0153] The coordinate mapping module 150 includes:

[0154] Camera calibration: a multi-pose checkerboard calibration method is used to establish a composite distortion model;

[0155] Hand-eye calibration: the conversion relationship between the camera coordinate system and the robot base coordinate system of the multi-coordinate mapping relationship is established, the calibration trajectory is generated by robot multi-axis linkage, and the conversion matrix is optimized based on kinematic constraints.

[0156] The six-degree-of-freedom pose instruction is transmitted in real time through an industrial Ethernet protocol, the industrial Ethernet protocol uses an EtherCAT protocol, a priority queue is set to guarantee real-time transmission of robot action instructions, and CRC check is used to ensure data integrity.

[0157] As shown in Figure 5 The hand-eye system coordinate relationship diagram of the running logic of the coordinate mapping module 150, the binocular camera system calibration and coordinate conversion are based on the principle of perspective projection, and the sensor of the camera obtains an image by sensing the light and shadow collected by the lens. From the image obtained by the camera shooting to the identification of the object guiding the robot to grab, the relationship conversion between multiple coordinate systems is involved.

[0158] 1. Camera calibration aspect

[0159] In the coordinate mapping module 150, a multi-pose checkerboard calibration method is used to establish a composite distortion model. Because in actual visual processing, the camera lens often has distortion and other problems, through the mature and effective technology of multi-pose checkerboard calibration method, the internal and external parameters of the camera can be accurately obtained, and an accurate composite distortion model can be established, thereby laying a foundation for subsequent accurate coordinate conversion and other operations.

[0160] 2. Hand-eye calibration aspect

[0161] When establishing the conversion relationship between the camera coordinate system and the robot base coordinate system of the multi-coordinate mapping relationship, a calibration trajectory is generated by robot multi-axis linkage. Here, robot multi-axis linkage refers to the generation of a specific trajectory by the coordinated movement of multiple joint axes of the robot, and the conversion matrix is optimized based on kinematic constraints. In this way, the kinematic characteristics of the robot itself can be fully utilized to accurately determine the conversion relationship between the camera coordinate system and the robot base coordinate system, making the coordinate conversion in different coordinate systems more accurate.

[0162] The module first rigidly aligns the world coordinate system of the three-dimensional reconstruction model with the robot base coordinate system. The robot in the world coordinate system (generally the world coordinate system coincides with the robot base coordinate system) can perform operations such as grasping the target object. Because the motion control of the robot is usually based on the world coordinate system or its own base coordinate system, through the above accurate coordinate conversion, the robot can accurately obtain the position and other information of the target object in its own coordinate system, thereby realizing precise grasping and other actions.

[0163] The industrial Ethernet protocol adopts the EtherCAT protocol, which sets up a priority queue to ensure real-time transmission of robot action instructions. EtherCAT protocol is a high-performance industrial Ethernet technology that can achieve high-precision synchronization and real-time data transmission through its unique distributed clock mechanism. In this system, by setting up a priority queue, the robot's action instructions and other key data can be transmitted quickly and accurately, avoiding problems such as data transmission delay affecting the normal work of the robot. At the same time, CRC (Cyclic Redundancy Check) is used to ensure data integrity. CRC is a common data verification method that generates a check code during data transmission and verifies it at the receiving end, allowing errors that may occur during data transmission to be detected and corrected in a timely manner, ensuring the reliability of the entire system data transmission.

[0164] Compared with traditional technology, traditional coordinate mapping methods rely on single calibration posture or static trajectory generation strategy, and have defects such as low calibration accuracy, poor dynamic scene adaptability, and data transmission delay. For example, traditional camera calibration uses a fixed posture checkerboard, which is difficult to correct lens composite distortion, resulting in accumulation of three-dimensional coordinate mapping error; hand-eye calibration relies on artificial preset trajectory and cannot fully utilize the kinematic characteristics of the robot to optimize the conversion matrix; ordinary Ethernet protocol lacks real-time guarantee mechanism and is easily affected by network delay or data packet loss, affecting the action accuracy of the robot. The present scheme solves the problems of large coordinate mapping error, insufficient real-time performance, and low data reliability in traditional methods through multi-posture checkerboard calibration, robot multi-axis linkage trajectory optimization, and industrial-grade Ethernet protocol collaborative design, significantly improving the operation accuracy and stability of the industrial automation system.

[0165] By the technical solution, the application realizes high-precision coordinate mapping and real-time control of the industrial robot vision guidance system. The coordinate mapping module accurately models camera distortion parameters through multi-pose calibration, dynamically optimizes the hand-eye conversion matrix in combination with the multi-axis linkage trajectory of the robot, and ensures sub-millimeter level mapping precision from three-dimensional space coordinates to the robot base coordinate system. The industrial Ethernet protocol based on the priority queue mechanism and CRC check technology of EtherCAT ensures real-time transmission and data integrity of the pose instruction. The modules interact closely based on a unified space-time reference, realize seamless connection from visual perception to mechanical arm action in complex industrial scenes, provide a high-reliability control foundation for part grabbing, precision assembly and other operations, and significantly improve the response speed and execution consistency of the automation system.

[0166] As shown in Figure 6 The point cloud data rapid processing system based on 3D vision also includes a dynamic parameter adjustment unit 160: according to the point cloud density and scene complexity, the dimension after dimension reduction and the registration iteration number are adaptively adjusted; the dynamic parameter adjustment unit switches to the random sampling dimension reduction mode when the point cloud density is lower than the set threshold according to the real-time processing delay, so as to improve the processing speed.

[0167] The dynamic parameter adjustment unit 160 adaptively adjusts the dimension after dimension reduction and the registration iteration number according to the point cloud density and scene complexity. The point cloud density refers to the number of points contained in a unit volume in three-dimensional space, which reflects the density of point cloud data. The scene complexity is a comprehensive measure of the distribution, shape and other complex characteristics of various objects in the entire three-dimensional scene.

[0168] The specific working mechanism of the dynamic parameter adjustment unit 160 is as follows: it continuously monitors the real-time processing delay. When the point cloud density is lower than the pre-set threshold, in order to effectively improve the overall processing speed, it automatically switches to the random sampling dimension reduction mode. In this random sampling dimension reduction mode, high-dimensional point cloud data is sampled through a specific random sampling algorithm, thereby reducing the data volume to some extent and speeding up the subsequent processing process.

[0169] Through the setting of the dynamic parameter adjustment unit 160, the entire system can flexibly adjust its related parameters in the face of different point cloud densities and scene complexities, so as to achieve better processing effect and higher processing efficiency, and ensure the stability and reliability of the system.

[0170] Compared with the traditional technology, the traditional point cloud processing system relies on fixed dimension reduction parameters and static registration strategy, and has defects such as poor scene adaptability and waste of computing resources. For example, the fixed dimension reduction algorithm is difficult to balance the precision loss of high-density point cloud and the data redundancy of low-density point cloud; the static iteration number setting is easy to cause excessive calculation in simple scene or convergence failure in complex scene. The scheme realizes real-time sensing of point cloud density and scene complexity through a dynamic parameter adjustment unit, adaptively switches the dimension reduction mode and optimizes the registration strategy, solves the technical bottleneck that the parameters are rigid, and the efficiency and precision are difficult to balance in the traditional method, and significantly improves the resource utilization and response capability of the system in dynamic industrial environment.

[0171] Through the above technical scheme, the application realizes dynamic optimization processing and efficient resource allocation of industrial point cloud data. The dynamic parameter adjustment unit selects the dimension reduction mode and the number of registration iterations by monitoring the point cloud density and processing delay in real time, greatly compresses the data size on the premise of ensuring feature integrity; the random sampling dimension reduction strategy quickly simplifies the data amount for sparse point cloud scene, combined with the dynamic threshold contraction mechanism of the registration module, realizes accurate pose alignment from coarse to fine. Based on the scene characteristics, each module deeply cooperates, balances the processing speed and accuracy demand in complex stacking, high dynamic interference and other industrial scenes, provides high-reliability real-time data support for robot grabbing, quality detection and other core processes, and comprehensively improves the environmental adaptability and operation stability of the automatic system.

[0172] As shown in Figure 6 The point cloud data rapid processing system based on 3D vision of the application further includes a safety protection unit 170, which realizes collision protection in the following way:

[0173] In the robot motion path planning, a virtual force field model is constructed based on the real-time pose of each joint of the robot arm, and the minimum interference distance between the robot arm and the obstacle is dynamically calculated;

[0174] When the interference potential of any joint exceeds the safety threshold, an emergency brake is triggered and an obstacle avoidance path re-planning instruction is generated;

[0175] The interference potential calculation comprehensively considers the weighted relationship between joint motion speed and environmental obstacle distance, realizing dynamic safety protection.

[0176] First, in the robot motion path planning link, a virtual force field model will be constructed based on the real-time pose of each joint of the robot arm. The virtual force field model here can be understood as a kind of force field-like environment simulated by a mathematical model, which can be constructed according to the current position and attitude of each joint of the robot arm and other information. Through the model, the minimum interference distance between the robot arm and the obstacle can be dynamically calculated. This calculation of the minimum interference distance is carried out in real time and is constantly updated with the movement of the robot arm and the changes in the position of the obstacles in the surrounding environment, so as to accurately grasp the proximity of the robot arm and the obstacles.

[0177] Secondly, when the interference potential energy of any joint of the robot arm exceeds the pre-set safety threshold, an emergency braking operation will be triggered. The interference potential energy here is not simply determined by a single factor, but is calculated by comprehensively considering the weighted relationship between the joint motion speed and the distance from the environment obstacle.

[0178] The formula of the interference potential energy calculation model is as follows:

[0179]

[0180] Parameter meaning:

[0181] U: interference potential energy, comprehensive motion speed and obstacle distance;

[0182] v: joint motion speed;

[0183] d: minimum distance from joint to obstacle;

[0184] α,β: weight coefficients, set according to safety requirements (e.g. α = 0.6, β = 0.4, experimental verification or empirical value).

[0185] For example, the faster the joint motion speed, the higher the interference potential energy may be in the case of the same distance from the obstacle; similarly, the closer the distance from the obstacle, even if the joint motion speed is slow, the interference potential energy may reach the level of triggering emergency braking. At the same time of triggering emergency braking, an obstacle avoidance path re-planning instruction will be generated. The generation of this instruction is to let the robot re-plan a reasonable motion path under the premise of avoiding collision, so that it can continue to complete the corresponding task or operation.

[0186] Through the above minimum interference distance calculation based on the virtual force field model, the interference potential energy calculation considering multiple factors, and the generation of the corresponding emergency braking and obstacle avoidance path re-planning instruction and other series of operations, the safety protection unit 170 realizes the dynamic safety protection of the robot system, which can effectively avoid the collision between the robot arm and the obstacles and other dangerous situations in the movement process, and ensure the safe and stable operation of the whole robot system.

[0187] Examples:

[0188] Test scenario: KUKA KR 1000 Titan moves at a speed of 1.5 m / s in a 2m x 2m x 2m space, with 5 dynamic obstacles randomly arranged.

[0189] Experimental results:

[0190]

[0191] Compared with the traditional technology, the traditional robot safety protection method depends on static distance threshold or single motion parameter monitoring, and has defects such as response delay, high risk of misjudgment, poor adaptability to dynamic scene, etc. For example, the fixed distance threshold cannot distinguish the risk difference between high-speed motion and low-speed approach scene, and is easy to cause excessive braking or collision detection missing; single speed or position monitoring ignores the influence of multi-factor coupling, and it is difficult to accurately evaluate the actual collision risk. The scheme solves the problems of rigid protection strategy and insufficient multi-factor collaborative judgment in the traditional method by constructing the environment force field in real time through the virtual force field model, and combining the interference potential to dynamically calculate the comprehensive risk of joint speed and obstacle distance, which significantly improves the safety and decision accuracy of the robot system in complex dynamic environment.

[0192] Through the above technical solutions, the real-time dynamic safety protection and intelligent obstacle avoidance control of the motion process of the industrial robot are realized. The virtual force field model dynamically calculates the minimum interference distance based on the pose of the robot arm and the distribution of obstacles, and maps the environmental risk in real time; the interference potential model quantifies the potential collision risk of joint motion by weighted fusion of speed and distance, accurately triggers emergency braking and generates an obstacle avoidance path. The safety protection unit and the path planning module deeply cooperate to effectively suppress the sudden collision risk while ensuring the motion efficiency of the robot arm, and ensure the operation continuity in complex stacked and multi-obstacle scenes. Based on the unified risk assessment framework, the modules interact closely to provide high-reliability safety protection for the automated production line, and significantly improve the environmental adaptability and running stability of the robot system.

[0193] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A 3D vision based point cloud data fast processing system, characterized in that, The method comprises the steps of: a binocular acquisition module (100) is used for acquiring multi-view images of target objects and generating high-precision three-dimensional point cloud data; a point cloud preprocessing module (110) is used for noise filtering and dimension reduction processing of the original point cloud data, specifically including: the noise filtering is based on neighborhood statistical filtering and voxel grid filtering to remove outliers and perform uniform sampling; the dimension reduction processing adopts a principal component analysis (PCA) algorithm for dimension reduction to retain main direction features; a hierarchical feature extraction module (120) is used for feature extraction of the preprocessed point cloud data by using an improved ORB algorithm to generate point cloud feature descriptors, specifically including: a multi-scale Gaussian pyramid is constructed to generate a scale-invariant feature detection space through layer-by-layer downsampling; curvature weight and non-maximum suppression are fused for feature screening to retain feature points; in the fusion of curvature weight and non-maximum suppression for feature screening, the non-maximum suppression preferentially retains edge points and feature points with significant curvature changes in key point detection, and abnormal points are removed through curvature consistency verification redundant feature points are removed through spatial distribution optimization; an industrial scene segmentation module (130) is based on normal vector clustering and region growing algorithm to simultaneously separate stacked object point clouds, and feedback correction is performed on the point cloud feature descriptors to generate corrected point cloud data; the normal vector clustering is based on local normal vector distribution of the point cloud to construct spatial clustering, and the spectral clustering algorithm is used to divide adjacent points with consistent normal vector directions of the point cloud feature descriptors into initial sub-regions for pre-segmentation; the region growing algorithm takes the initial sub-regions as growing seeds, and iteratively expands the region boundary in combination with the following conditions: a dynamic threshold for the consistency of the normal vector directions of adjacent points, which is adjusted adaptively according to the local surface curvature change rate; the deviation between the region growing direction and the main curvature direction is not more than a preset angle threshold; the industrial scene segmentation module (130) performs conflict detection on overlapping regions, selects the optimal segmentation boundary based on curvature continuity and spatial connectivity, and performs secondary verification on controversial regions through a lightweight semantic segmentation model; the lightweight semantic segmentation model is a point cloud segmentation network based on an attention mechanism, which is adapted to different industrial scene object categories through transfer learning; a two-stage registration module (140) sequentially performs a coarse registration stage and a fine registration stage on the corrected point cloud data to generate a three-dimensional reconstruction model; the coarse registration stage selects matching point pairs based on ORB feature matching and RANSAC algorithm, and calculates an initial rotation matrix and a translation vector; the fine registration stage adopts an improved ICP algorithm, combines KD-Tree to accelerate nearest neighbor search, and is optimized through a dynamic distance threshold; the distance threshold optimization includes: dynamically adjusting the distance threshold for corresponding point selection according to the weighted relationship between the maximum point distance of the current point cloud frame and the registration error residual; in the KD-Tree acceleration search process, the normal vector consistency constraint is introduced, and only the nearest neighbor point pairs with a normal vector included angle less than the set distance threshold are retained; through an iterative error feedback mechanism, the distance threshold range is gradually contracted to approach the optimal registration accuracy. The coordinate mapping module (150) establishes a multi-coordinate system mapping relationship based on the three-dimensional reconstruction model, outputs a six-degree-of-freedom pose instruction, and transmits in real time; The dynamic parameter adjustment unit (160) adaptively adjusts the dimension after dimension reduction and the iteration number of registration according to the point cloud density and the scene complexity; the dynamic parameter adjustment unit switches to a random sampling dimension reduction mode when the point cloud density is lower than a set threshold according to the real-time processing delay, so as to improve the processing speed.

2. The system for fast processing of point cloud data based on 3D vision according to claim 1, wherein, The neighborhood statistical filtering dynamically removes noise points exceeding a preset threshold by calculating the standard deviation of the neighborhood distance of the point cloud; the voxel grid filtering uniformly down-samples at an adaptive resolution. 3.The 3D vision-based point cloud data fast processing system of claim 1, wherein, The coordinate mapping module (150) comprises: Camera calibration: a multi-pose checkerboard calibration method is used to establish a composite distortion model; Hand-eye calibration: the conversion relationship between the camera coordinate system and the robot base coordinate system of the multi-coordinate system mapping relationship is established, a calibration trajectory is generated through robot multi-axis linkage, and the conversion matrix is optimized based on kinematic constraints.

4. The system of claim 1, wherein, The six-degree-of-freedom pose instruction is transmitted in real time through an industrial Ethernet protocol, the industrial Ethernet protocol adopts an EtherCAT protocol, a priority queue is set to guarantee real-time transmission of robot action instructions, and CRC check is used to ensure data integrity.

5. The system for fast processing of point cloud data based on 3D vision of claim 1, wherein, The system further comprises a safety protection unit (170) which realizes collision protection in the following manner: In the robot motion path planning, a virtual force field model is constructed based on the real-time pose of each joint of the robot arm, and the minimum interference distance between the robot arm and the obstacle is dynamically calculated; When the interference potential of any joint exceeds a safety threshold, emergency braking is triggered and an obstacle avoidance path re-planning instruction is generated; The interference potential calculation comprehensively considers the weighted relationship between the joint motion speed and the distance to the environmental obstacle, so as to realize dynamic safety protection.

Images