A mechanical arm welding environment perception method and system based on a dual photoelectric frequency comb and a laser radar

Abstract

The application discloses a kind of based on double photoelectric frequency comb and laser radar mechanical arm welding environment perception method and system, including data acquisition module, data pre-processing module, ground segmentation module, projection conversion module, clustering processing module, semantic identification module and result output module.The data acquisition module is by depth integrated common light path or tight coupling optical design, so that double photoelectric frequency comb provides the absolute distance reference of tracing to atomic clock and real-time error correction for laser radar point cloud.Point cloud processing algorithm is optimized for welding scene, including introducing the improved RANSAC ground segmentation of normal vector constraint, projection guided three-dimensional density clustering and lightweight semantic identification based on geometric features.The application effectively solves the problem of decreased perception quality caused by high temperature, arc light and smoke interference in welding environment, supports the realization of adaptive welding parameter adjustment and active obstacle avoidance control for mechanical arm, and has high precision, high robustness and real-time performance.

Description

Technical Field

[0001] This invention belongs to the fields of advanced manufacturing technology, lidar sensing, optical frequency comb measurement and industrial robot control. Specifically, it relates to a robotic arm welding environment perception method and system based on dual photoelectric frequency combs and lidar, which is used for real-time three-dimensional reconstruction, obstacle recognition and adaptive control during the welding process. Background Technology

[0002] With the development of industrial automation, robotic welding is widely used in automobile manufacturing, shipbuilding, and other fields. However, traditional welding robotic arms rely on preset paths and lack the ability to adapt to dynamic environments. Existing environmental perception technologies are mainly based on visual sensors or two-dimensional LiDAR, which have the following limitations:

[0003] Visual sensors are affected by welding arc light, smoke, and spatter, resulting in decreased image quality and an obstacle recognition accuracy of less than 60%.

[0004] Traditional LiDAR point cloud data is sparse and noisy, susceptible to high temperatures and electromagnetic interference, and clustering algorithms suffer from oversegmentation or undersegmentation errors exceeding 10%. Visual sensors are affected by welding arc light, smoke, and spatter, resulting in degraded image quality and obstacle recognition accuracy of less than 60%. Furthermore, the sparse and noisy nature of traditional LiDAR point cloud data, coupled with its susceptibility to high temperatures and electromagnetic interference, makes oversegmentation or undersegmentation during point cloud clustering prone to occur, leading to unstable obstacle boundary extraction and impacting the reliability of subsequent target recognition and robotic arm obstacle avoidance control. Existing methods only output geometric clusters, lacking semantic information and unable to distinguish between workpieces, tools, or personnel, hindering intelligent control.

[0005] Existing methods only output geometric clusters, lacking semantic information, and cannot distinguish between workpieces, tools, or personnel, making it difficult to achieve intelligent control.

[0006] While dual-photoelectric frequency comb technology offers advantages in high-precision measurement, it has not yet been deeply integrated with lidar point cloud processing to solve the challenges of welding environments. Therefore, a high-precision sensing method that integrates multi-source data is urgently needed. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a real-time perception and control system for robotic arm welding based on dual-photoelectric frequency comb enhanced lidar. The core idea of ​​this invention is to use the dual-photoelectric frequency comb as a "ruler" to provide a high-precision absolute distance reference and real-time error correction for lidar point cloud data, thereby significantly improving the quality of the point cloud data. Combined with point cloud processing and semantic understanding algorithms specifically designed for welding scenarios, this ultimately achieves ultra-precise and robust real-time perception of the welding environment, driving the robotic arm to complete adaptive and intelligent welding operations.

[0008] The objective of this invention is achieved through the following technical solution: a robotic arm welding environment perception system based on dual photoelectric frequency combs and lidar, the system comprising:

[0009] The data acquisition module is used to generate three-dimensional point cloud data by scanning the welding environment with lidar, and integrates a dual photoelectric frequency comb device to calibrate the point cloud depth in real time through a common optical path or tightly coupled optical design.

[0010] The data preprocessing module is used to filter and fuse the point cloud after frequency comb calibration;

[0011] The ground segmentation module is used to separate ground and non-ground points using an improved RANSAC algorithm that incorporates normal vector constraints and iterative optimization mechanisms.

[0012] The projection conversion module is used to project non-ground point clouds onto a two-dimensional depth image;

[0013] The clustering module is used to employ a projection-guided strategy to first perform connected region analysis and locate seed points in the two-dimensional image, and then backtrack to the three-dimensional space to perform DBSCAN clustering.

[0014] A semantic recognition module is used for lightweight classification based on the geometric features of point cloud clusters;

[0015] The results output module is used to generate visual results and drive the control of the robotic arm.

[0016] Furthermore, in the data acquisition module, the common-path or tightly coupled optical design includes a common-path design or a tightly coupled design. The common-path design involves combining the lidar measurement beam and the dual-electro-optical frequency comb measurement beam into a coaxial or near-coaxial beam using a dichroic mirror, polarization combiner, or fiber coupler. This beam then illuminates the welding area through the same scanning window, and the returning light is split by a beam splitter and enters the lidar receiver and the dual-electro-optical frequency comb interferometric measurement end, respectively. The tightly coupled design involves fixing the lidar measurement unit and the dual-electro-optical frequency comb ranging unit on the same rigid mounting base. The spatial extrinsic parameter relationship between the two is obtained through a calibration plate or standard target, establishing a correspondence between the dual-electro-optical frequency comb ranging result and the lidar point cloud data.

[0017] Furthermore, in the data acquisition module, the dual photoelectric frequency comb emits two beams of optical frequency comb light with a precise fixed frequency difference. The flight time or phase of the lidar beam is calibrated in real time by interferometry, and the system error caused by atmospheric turbulence, temperature change, and laser jitter is dynamically corrected.

[0018] Furthermore, the ground segmentation module employs an improved RANSAC algorithm to separate ground points from non-ground points. After each generation of a candidate ground plane, the normal vector of the candidate ground plane is compared with a preset ground reference normal vector or the ground normal vector of the previous frame. If the included angle is less than a preset slope angle threshold, the candidate ground plane is retained; otherwise, it is discarded. When determining inliers, ground points are determined based on the vertical distance from the point to the candidate ground plane and the consistency of the point's local normal vector. The distance threshold is dynamically adjusted based on the current point cloud noise level and the distance residual from the point to the candidate ground plane, and the maximum number of iterations is dynamically adjusted based on the proportion of inliers in the candidate ground plane.

[0019] Furthermore, the clustering processing module dynamically optimizes the neighborhood radius and minimum number of points parameters of the DBSCAN algorithm based on the point cloud density, welding speed, and current interference level to ensure stable clustering results under different working conditions.

[0020] Furthermore, the semantic recognition module classifies point cloud clusters according to preset rules based on their three-dimensional size, volume, aspect ratio, convex hull shape, and height position features.

[0021] Furthermore, the system also includes a multimodal data fusion architecture, which ensures strict synchronization between point cloud and frequency comb data through a spatiotemporal synchronization module with high-precision hardware triggering and unified timestamp, and adopts a hierarchical fusion strategy, including point-level fusion, feature-level fusion, and an adaptive weight allocation mechanism based on real-time signal-to-noise ratio. In areas with strong arc light interference, frequency comb data is used preferentially to improve the signal-to-noise ratio.

[0022] Furthermore, the system also includes a multimodal data fusion architecture, which includes a spatiotemporal synchronization module and a data fusion module. The spatiotemporal synchronization module is used to synchronize the lidar point cloud data and the dual-electro-optical frequency comb ranging data through a hardware trigger signal and a unified timestamp. The data fusion module is used to convert the dual-electro-optical frequency comb ranging data to the lidar coordinate system according to the calibration relationship between the lidar coordinate system and the dual-electro-optical frequency comb ranging coordinate system, and to correct the point cloud depth value, geometric features and target recognition results of the corresponding area.

[0023] Furthermore, the result output module sends the semantic recognition results to the robotic arm control system in real time via JSON format. The system drives the adaptive adjustment of welding parameters and collision risk prediction and obstacle avoidance strategies based on the identified target type.

[0024] On the other hand, the present invention also provides a robotic arm welding environment perception method based on dual photoelectric frequency combs and lidar, the method comprising the following steps:

[0025] (1) Generate three-dimensional point cloud data by scanning the welding environment with lidar, and integrate dual photoelectric frequency comb equipment to calibrate the point cloud depth in real time through common optical path or tight coupling optical design;

[0026] (2) Filtering and fusing the point cloud after frequency comb calibration;

[0027] (3) An improved RANSAC algorithm with introduced normal vector constraints and iterative optimization mechanism is used to separate ground and non-ground points;

[0028] (4) Project the non-ground point cloud onto the two-dimensional depth image;

[0029] (5) Using a projection-guided strategy, first perform connected region analysis to locate seed points in the two-dimensional image, and then backtrack to the three-dimensional space to perform DBSCAN clustering;

[0030] (6) Lightweight classification based on the geometric features of point cloud clusters;

[0031] (7) Generate visualization results and drive the robotic arm control.

[0032] The beneficial effects of this invention are:

[0033] Dual-electro-frequency comb enhanced lidar system: Through innovative common-path or tightly coupled optical system design, the lidar's measurement beam and the probe beam of the dual-electro-frequency comb are highly overlapped in space. The frequency comb calibrates the lidar beam in real time using interferometry, dynamically correcting system errors and improving point cloud accuracy from centimeter-level to sub-millimeter-level. Simultaneously, the frequency comb provides absolute distance traceability to the atomic clock, enabling high-precision 3D scanning of the workpiece to create a digital model before welding.

[0034] A dedicated point cloud processing algorithm for welding scenarios: This algorithm comprehensively optimizes the point cloud processing pipeline. Ground segmentation employs an improved RANSAC algorithm, introducing normal vector constraints and iterative optimization mechanisms to adapt to the slope of the workshop floor. Clustering processing innovatively adopts a projection-guided strategy, first projecting the point cloud onto a 2D depth image for rapid connected region analysis, then backtracking to 3D space for refined DBSCAN clustering, and dynamically optimizing parameters based on working conditions to effectively reduce over-segmentation and under-segmentation.

[0035] Multimodal data fusion architecture: A dedicated spatiotemporal synchronization module and multi-scale fusion algorithm were designed. Data synchronization is ensured through high-precision hardware triggering and unified timestamps. Point-level fusion (correcting key point coordinates), feature-level fusion (improving the accuracy of geometric features), and adaptive weight allocation mechanism are adopted to improve the signal-to-noise ratio in areas with strong interference.

[0036] Real-time semantic understanding and decision-making system: A lightweight classifier based on rules and simple machine learning was designed to identify target types (workpieces, personnel, tools, etc.) in real time using the geometric features of point cloud clusters (such as size, aspect ratio, and height). Semantic information is directly used to drive the adaptive adjustment of welding process parameters and collision risk prediction and obstacle avoidance strategies based on linear prediction algorithms, ensuring the safety of human-machine collaboration. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 It is a system architecture diagram that shows the complete process from data acquisition and processing to output;

[0039] Figure 2 This is a top-view ground segmentation result, showing the improved RANSAC algorithm's ability to separate ground and non-ground points;

[0040] Figure 3 It is a projected depth image (ground = 0), showing the result of projecting a point cloud onto a two-dimensional depth map;

[0041] Figure 4 This is the clustering result (example: 90 clusters), showing the distribution of point cloud clusters after DBSCAN clustering;

[0042] Figure 5 It is a histogram of cluster size distribution, which statistically analyzes the range of cluster sizes. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.

[0044] like Figure 1 As shown, this embodiment of the invention uses the welding of automotive parts by a robotic arm as an example for illustration. The specific implementation process is as follows:

[0045] 1. System Deployment and Data Acquisition: The robotic arm welding environment perception system based on dual photoelectric frequency combs and lidar provided by this invention is deployed in the welding workshop. A lidar (such as a Velodyne or Ouster model) is installed at the end of the robotic arm or at a fixed position in the working area to scan the dynamic welding environment and generate a 3D point cloud. The dual photoelectric frequency comb device and the lidar work together using a shared optical path design, synchronously emitting light pulses to perform high-precision distance calibration on key areas such as welding points.

[0046] In this embodiment, the common optical path or tightly coupled optical design can be implemented in the following two ways.

[0047] One approach involves a common-path design: the dual-electro-optical frequency comb measurement beam, after collimation, is combined with the lidar measurement beam via a dichroic mirror, polarization combiner, or fiber coupler, ensuring that both beams illuminate the welding area along the same optical axis or a substantially identical spatial path. The reflected light from the target then passes through a beam-splitting assembly, entering the lidar receiver and the dual-electro-optical frequency comb interferometric measurement end, respectively. Thus, the high-precision distance value measured by the dual-electro-optical frequency comb corresponds to the lidar point cloud depth value in the same measurement direction.

[0048] Another approach is a tightly coupled design: when the lidar optical path and the dual photoelectric frequency comb optical path are not completely coaxial, they are fixedly mounted on the same rigid mounting base, and their spatial extrinsic parameter relationship is pre-calibrated using a standard target or calibration plate. Simultaneously, data synchronization is achieved through a hardware trigger signal or a unified timestamp. Based on this spatial extrinsic parameter relationship, the system converts the dual photoelectric frequency comb ranging results to the lidar coordinate system to correct the point cloud depth error in the corresponding area. Through the above-mentioned common optical path or tightly coupled design, the dual photoelectric frequency comb can provide a high-precision distance reference for the lidar, thereby improving the measurement accuracy and stability of point cloud data in the welding environment.

[0049] 2. Data Preprocessing and Fusion: In the data preprocessing module, the depth values ​​of the LiDAR point cloud are first calibrated in real time using absolute distance reference data provided by dual photoelectric frequency combs to correct errors introduced by environmental factors. Subsequently, voxel grid downsampling is applied to the calibrated point cloud to reduce the amount of data while preserving key features, thus meeting real-time requirements.

[0050] 3. Point cloud processing and semantic recognition:

[0051] Ground segmentation: An improved RANSAC algorithm is used to process the point cloud. Specifically, the system first performs a neighborhood search on the point cloud to calculate the local normal vector of each point; then, after generating candidate ground planes through RANSAC random sampling, the normal vector of the candidate plane is compared with the preset ground reference normal vector or the ground normal vector estimated in the previous frame. If the angle between the two is less than a preset slope angle threshold, the candidate plane is retained; if the angle is too large, the candidate plane is discarded to avoid misidentifying non-ground structures such as workpiece sides and robot arm surfaces as ground. In the interior point determination stage, ground points are simultaneously determined based on the vertical distance from the point to the candidate ground plane and the consistency of the local normal vector. For points... Its path to the candidate plane The distance is:

[0052]

[0053] when If the distance is less than a set distance threshold and the local normal vector of the point is aligned with the normal vector of the candidate plane, it is classified as a ground point; otherwise, it is classified as a non-ground point. The distance threshold is the vertical distance from the point to the candidate ground plane. Furthermore, the system dynamically adjusts the distance threshold and the number of RANSAC iterations based on the point cloud noise level and the proportion of points within the ground: the distance threshold is appropriately widened when the point cloud noise is high, and the distance threshold is reduced when the point cloud quality is good; the number of iterations is reduced when the proportion of points within the ground is high, and the number of iterations is increased when there is significant ground occlusion or slope change. This allows for the stable separation of workshop ground points from non-ground points such as workpieces, personnel, and obstacles. Figure 2 It demonstrates the effect of ground segmentation.

[0054] Projection transformation: Projecting non-ground point clouds onto a 2D front view depth image with dimensions of 65 (height) x 2049 (width) pixels, such as... Figure 3 As shown, the three-dimensional clustering problem is transformed into a more efficient two-dimensional image analysis.

[0055] Clustering Processing: Projection-guided DBSCAN clustering is employed. First, connected component analysis is performed on the 2D depth map to quickly locate seed point regions. Then, density clustering is performed in 3D space. Clustering parameters (such as neighborhood radius and minimum number of points) are dynamically optimized based on the current point cloud density and welding speed to ensure clustering stability. Figure 4 An example of clustering results (90 clusters) is shown.

[0056] Semantic recognition: Classification is performed based on the geometric features (height, number of points, volume, aspect ratio, etc.) of each point cloud cluster. For example, clusters with a height of less than 1.0 meter and fewer than 100 points are identified as "small tools" or "miscellaneous items", clusters with a height of less than 2.0 meters and fewer than 500 points are identified as "personnel" or "mobile equipment", clusters with a height of around 3.0 meters and around 2000 points are identified as "welded workpieces", and clusters with a height greater than 4.0 meters are classified as "large structures". Figure 5 The cluster size distribution provides a basis for parameter optimization.

[0057] 4. Output and Control: The semantic recognition results (including target type, location, size, etc.) are sent to the robotic arm control system in real time via JSON format. The system makes decisions based on this: when a "weld" is detected, the welding current, speed, and other parameters are adaptively adjusted; when a "person" is detected approaching, a collision risk prediction algorithm is triggered to perform obstacle avoidance actions such as deceleration, pausing, or replanning the path.

[0058] Actual measurements show that the average latency for processing one frame of point cloud data is approximately 2.25 seconds, which meets the requirements for real-time control. In a typical welding workshop environment, the recognition rate for welded workpieces reaches 95.2%, and the recognition rate for personnel reaches 89.7%, significantly outperforming traditional sensing methods. This invention significantly improves the accuracy, robustness, and intelligence level of welding environment perception by deeply fusing dual photoelectric frequency combs and lidar, and optimizing the point cloud processing algorithm.

[0059] This invention significantly improves the accuracy and robustness of welding environment perception by integrating dual photoelectric frequency combs and lidar, thus promoting the development of industrial automation and intelligence.

[0060] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A robotic arm welding environment perception system based on dual photoelectric frequency combs and lidar, characterized in that: The system includes: The data acquisition module is used to generate three-dimensional point cloud data by scanning the welding environment with lidar, and integrates a dual photoelectric frequency comb device to calibrate the point cloud depth in real time through a common optical path or tightly coupled optical design. The data preprocessing module is used to filter and fuse the point cloud after frequency comb calibration; The ground segmentation module is used to separate ground and non-ground points using an improved RANSAC algorithm that incorporates normal vector constraints and iterative optimization mechanisms. The projection conversion module is used to project non-ground point clouds onto a two-dimensional depth image; The clustering module is used to employ a projection-guided strategy to first perform connected region analysis and locate seed points in the two-dimensional image, and then backtrack to the three-dimensional space to perform DBSCAN clustering. A semantic recognition module is used for lightweight classification based on the geometric features of point cloud clusters; The results output module is used to generate visual results and drive the control of the robotic arm.

2. The system according to claim 1, characterized in that: In the data acquisition module, the common-path or tightly coupled optical design includes a common-path design or a tightly coupled design. The common-path design involves combining the lidar measurement beam and the dual-electro-optical frequency comb measurement beam into a coaxial or near-coaxial beam using a dichroic mirror, polarization combiner, or fiber coupler. The beam then illuminates the welding area through the same scanning window, and the return light is split into the lidar receiver and the dual-electro-optical frequency comb interferometric measurement end, respectively. The tightly coupled design involves fixing the lidar measurement unit and the dual-electro-optical frequency comb ranging unit on the same rigid mounting base. The spatial extrinsic parameter relationship between the two is obtained through a calibration plate or standard target, establishing a correspondence between the dual-electro-optical frequency comb ranging result and the lidar point cloud data.

3. The system according to claim 1, characterized in that: In the data acquisition module, the dual photoelectric frequency comb emits two beams of optical frequency comb light with a precise fixed frequency difference. The flight time or phase of the lidar beam is calibrated in real time by interferometry, and the system error caused by atmospheric turbulence, temperature change, and laser jitter is dynamically corrected.

4. The system according to claim 1, characterized in that: The ground segmentation module uses an improved RANSAC algorithm to separate ground points from non-ground points. After each candidate ground plane is generated, the normal vector of the candidate ground plane is compared with the preset ground reference normal vector or the ground normal vector of the previous frame. If the angle is less than the preset slope angle threshold, the candidate ground plane is retained; otherwise, it is discarded. When determining inliers, ground points are determined based on the vertical distance from the point to the candidate ground plane and the consistency of the point's local normal vector. The distance threshold is dynamically adjusted based on the current point cloud noise level and the distance residual from the point to the candidate ground plane, and the maximum number of iterations is dynamically adjusted based on the proportion of inliers in the candidate ground plane.

5. The system according to claim 1, characterized in that: The clustering processing module dynamically optimizes the neighborhood radius and minimum number of points parameters of the DBSCAN algorithm based on the point cloud density, welding speed, and current interference level to ensure stable clustering results under different working conditions.

6. The system according to claim 1, characterized in that: The semantic recognition module classifies point cloud clusters according to preset rules based on their three-dimensional dimensions, volume, aspect ratio, convex hull shape, and height position features.

7. The system according to claim 1, characterized in that: The system also includes a multimodal data fusion architecture, which ensures strict synchronization between point cloud and frequency comb data through a spatiotemporal synchronization module with high-precision hardware triggering and unified timestamps. It also adopts a hierarchical fusion strategy, including point-level fusion, feature-level fusion, and an adaptive weight allocation mechanism based on real-time signal-to-noise ratio. In areas with strong arc light interference, frequency comb data is used first to improve the signal-to-noise ratio.

8. The system according to claim 1, characterized in that: The system also includes a multimodal data fusion architecture, which comprises a spatiotemporal synchronization module and a data fusion module. The spatiotemporal synchronization module is used to synchronize the lidar point cloud data and the dual-electro-optical frequency comb ranging data through a hardware trigger signal and a unified timestamp. The data fusion module is used to convert the dual-electro-optical frequency comb ranging data to the lidar coordinate system according to the calibration relationship between the lidar coordinate system and the dual-electro-optical frequency comb ranging coordinate system, and to correct the point cloud depth value, geometric features, and target recognition results of the corresponding area.

9. The system according to claim 1, characterized in that: The result output module sends the semantic recognition results to the robotic arm control system in real time via JSON format. The system drives the adaptive adjustment of welding parameters and collision risk prediction and obstacle avoidance strategies based on the identified target type.

10. An environmental perception method for a robotic arm welding environment perception system based on the dual photoelectric frequency comb and lidar as described in any one of claims 1-9, characterized in that: The method includes the following steps: (1) Generate three-dimensional point cloud data by scanning the welding environment with lidar, and integrate dual photoelectric frequency comb equipment to calibrate the point cloud depth in real time through common optical path or tight coupling optical design; (2) Filtering and fusing the point cloud after frequency comb calibration; (3) An improved RANSAC algorithm with introduced normal vector constraints and iterative optimization mechanism is used to separate ground and non-ground points; (4) Project the non-ground point cloud onto the two-dimensional depth image; (5) Using a projection-guided strategy, first perform connected region analysis to locate seed points in the two-dimensional image, and then backtrack to the three-dimensional space to perform DBSCAN clustering; (6) Lightweight classification based on the geometric features of point cloud clusters; (7) Generate visualization results and drive the robotic arm control.

Images