A workpiece completion method based on deep learning and surface estimation

Abstract

A workpiece completion method based on deep learning and surface estimation, relating to the field of workpiece inspection technology, includes: acquiring images and point clouds of the workpiece and fixture using an RGB camera and LiDAR; achieving coarse localization of the contact area between the workpiece and fixture through analysis; reconstructing the LiDAR point cloud to obtain an overall model and planning the shooting trajectory of a structured light camera; globally scanning along the trajectory to obtain a high-density point cloud, and reconstructing a fine point cloud model containing the workpiece and fixture; based on the coarse localization results, acquiring local data of the contact area using a structured light camera; performing fine semantic segmentation on the local data, removing the fixture point cloud to obtain the workpiece point cloud; performing surface fitting and uniform sampling on the workpiece point cloud to generate an estimated point cloud of the occluded area; and stitching the points together to obtain the complete workpiece point cloud. This invention solves the problems of inaccurate contact area segmentation and difficulty in completing occluded parts in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of workpiece inspection technology, and specifically to a workpiece completion method based on deep learning and surface estimation. Background Technology

[0002] In the fields of industrial manufacturing and automated inspection, workpieces typically need to be positioned and fixed using tooling or fixtures to ensure their stability during processing, assembly, or inspection. With the continuous improvement of intelligent manufacturing and industrial automation, higher precision and intelligence requirements are being placed on tasks such as 3D reconstruction of workpieces, posture recognition, surface quality inspection, and robot path planning. The completion of these tasks often depends on the accurate acquisition and segmentation of 3D data of the workpiece, especially the precise identification and separation of the contact area between the workpiece and the tooling.

[0003] Currently, common methods for acquiring 3D workpiece data include laser scanning, structured light imaging, and stereo vision. Among these, lidar scanning has the advantages of large-area and fast scanning, making it suitable for constructing the overall geometric framework of a scene. Structured light cameras, on the other hand, can achieve high-precision 3D reconstruction at close range, making them suitable for fine measurement of local details. However, a single sensor often struggles to meet both global and local accuracy requirements. Furthermore, in areas where the workpiece and tooling are in close contact, the complex geometry of the contact surface and severe occlusion make it difficult to achieve accurate segmentation by relying solely on a single type of sensor.

[0004] At the data processing level, traditional point cloud segmentation methods are mostly based on geometric rules or threshold judgments, such as attempting to separate different objects through curvature changes, normal vector differences, and distance thresholds. However, in the contact area between the workpiece and the tooling, due to the similar point cloud density and blurred geometric features, traditional methods are prone to missegmentation or omission, making it difficult to effectively distinguish between the two. In addition, although traditional segmentation methods based on RGB images can utilize semantic information for region division, the lack of depth information makes it difficult to accurately locate the contact boundary in three-dimensional space, and the registration error between the image and the point cloud further affects the segmentation accuracy.

[0005] In recent years, deep learning technology has made significant progress in image segmentation, point cloud processing, and other fields. Neural network-based semantic segmentation models can extract rich texture and semantic information from RGB images, providing prior support for semantic annotation of point clouds. However, existing methods mostly focus on single-modal data processing and lack effective fusion of multi-source sensor data, making it difficult to fully leverage the complementary advantages of different sensors in terms of accuracy, range, and semantic information. Especially in the task of fine segmentation of the contact area between the workpiece and the tooling, how to combine the global structural information of LiDAR with the high-precision local data of structured light cameras to achieve coarse-to-fine segmentation and completion remains a challenge in current technology.

[0006] In summary, existing technologies have the following shortcomings in the segmentation and completion of the contact area between the workpiece and the tooling: First, traditional geometric segmentation methods are difficult to handle complex contact surfaces; second, single sensor data cannot take into account both global and local accuracy; third, there is a lack of effective multimodal data fusion mechanisms; and fourth, the application of deep learning models in 3D segmentation is mostly limited to a single data source and fails to fully combine image semantics and high-precision depth information.

[0007] To address this, we propose a method that accurately segments the contact area between the workpiece and the tooling, and can complete the areas of the workpiece that are obscured. Summary of the Invention

[0008] The purpose of this invention is to provide a workpiece completion method based on deep learning and surface estimation, which solves the problems of inaccurate segmentation of the contact area between the workpiece and the tooling and the difficulty in completing the occluded parts in traditional methods.

[0009] This invention is achieved through the following technical solution: A workpiece completion method based on deep learning and surface estimation, specifically including: RGB cameras and LiDAR are used to acquire RGB images of the workpiece and tooling in the working area and LiDAR point clouds; Based on the analysis of RGB images and LiDAR point clouds, the contact area between the workpiece and the tooling is determined to achieve coarse positioning; Based on the lidar point cloud, a global point cloud model of the working area is reconstructed, and an accessibility analysis is performed on the global point cloud model to generate the shooting trajectory. A structured light camera is used to perform a global scan of the working area along the shooting trajectory to obtain a high-density point cloud and reconstruct it, resulting in a fine point cloud model that includes the workpiece and tooling. Based on the coarse positioning results, a structured light camera was used to collect local data of the contact area. Fine semantic segmentation is performed on local data, and the point cloud of the tooling part is removed from the fine point cloud model to obtain the point cloud of the workpiece part. Perform predictive estimation analysis on the point cloud of the workpiece to generate an estimated point cloud of the area occluded by the tooling. By stitching together the estimated point cloud and the point cloud of the workpiece portion, the completed workpiece point cloud is obtained.

[0010] Furthermore, the analysis based on RGB images and LiDAR point clouds detects the contact area between the workpiece and the tooling, achieving coarse positioning. The specific steps are as follows: A semantic segmentation network is used to segment RGB images into workpieces, fixtures, and background to obtain pixel-level semantic masks. The semantic mask and the lidar point cloud are registered to obtain the semantic point cloud. Coarse positioning is achieved by detecting the contact area between the workpiece and the tooling based on semantic point cloud.

[0011] Furthermore, the registration of the semantic mask and the LiDAR point cloud to obtain the semantic point cloud specifically involves: projecting the LiDAR point cloud onto the RGB image plane based on the extrinsic calibration parameters between the RGB camera and the LiDAR to obtain the pixel coordinates; By querying the category label of the semantic mask at pixel coordinates, the semantic label is associated and fused with the corresponding LiDAR point cloud to finally obtain the semantic point cloud.

[0012] Furthermore, the overall point cloud model of the working area reconstructed based on the lidar point cloud is analyzed for accessibility to generate the shooting trajectory. The specific steps are as follows: The lidar point cloud is stitched, denoised, filtered, and registered to reconstruct the overall point cloud model of the working area. Based on the semantic point cloud, the spatial distribution range of the workpiece is determined from the reconstructed overall point cloud model of the lidar, and the region of interest is delineated accordingly to determine the scanning range that the structured light camera needs to cover. The extracted workpiece contour region is subjected to surface normal vector estimation and curvature analysis to generate an initial set of viewpoints for covering the workpiece surface, wherein each viewpoint contains the preset spatial position and optical axis orientation of the structured light camera. By combining the robot's kinematic model and the obstacle distribution in the workspace, accessibility analysis and collision detection are performed on the initial set of viewpoints to eliminate viewpoints that the robotic arm cannot reach or that pose a collision risk. Based on the greedy projection optimization algorithm, the remaining available viewpoints are sorted and redundant points are removed to plan a smooth, continuous and complete structured light camera shooting trajectory.

[0013] Furthermore, when the structured light camera acquires local data of the contact area, the focal length is set to 300mm-500mm.

[0014] Furthermore, the local data includes local high-precision depth maps, grayscale images, and local high-density point clouds.

[0015] Furthermore, the specific steps for predicting and estimating the point cloud of the workpiece portion to generate an estimated point cloud of the area occluded by the tooling are as follows: Based on the point cloud of the workpiece, a non-uniform rational B-spline (NURBS) surface is fitted to the workpiece surface to generate a fitted surface. Perform uniform sampling based on the fitted surface; Substituting the sampling results into the surface equation yields a uniformly distributed estimated point cloud.

[0016] Furthermore, the non-uniform rational B-spline surface fitting in the parameter space The above is performed, and the specific expression is:

[0017] In the formula, As control points, As weight, and They are respectively order and B-spline basis functions of order 1 and This represents the number of control points in the two parameter directions.

[0018] Furthermore, the objective function for fitting the non-uniform rational B-spline surface is:

[0019] In the formula, The first point cloud of the workpiece A point cloud, for The corresponding parameter values, This represents the total number of point clouds in the workpiece section.

[0020] Furthermore, the uniform sampling based on the fitted surface specifically refers to the sampling in the parameter space. The calculation is performed internally, and the formula is as follows:

[0021]

[0022] and:

[0023]

[0024] In the formula, and For in parameter space direction and The number of points generated in the direction.

[0025] The technical solution of the present invention has at least the following advantages and beneficial effects: This invention discloses a workpiece completion method based on deep learning and surface estimation. By acquiring multi-source data from RGB cameras and LiDAR, and combining deep learning semantic segmentation and point cloud registration techniques, coarse localization of the contact area between the workpiece and the tooling is achieved, solving the problem that traditional geometric segmentation methods cannot handle complex contact surfaces. At the same time, a structured light camera is used to perform local high-precision acquisition and fine semantic segmentation of the contact area, which can accurately separate the point clouds of the workpiece and the tooling, significantly improving the segmentation accuracy of the contact area.

[0026] In addition, a hierarchical scanning strategy of global coarse scanning by lidar and local fine scanning by structured light camera is adopted. This strategy not only utilizes the advantage of lidar's wide coverage to construct the overall geometric framework, but also leverages the advantage of structured light camera's high-precision reconstruction to obtain local details, thus overcoming the limitation of a single sensor being unable to simultaneously achieve global and local accuracy.

[0027] Furthermore, by fusing and segmenting image semantic information with point cloud data, the semantic understanding capability of deep learning in 2D images is introduced into 3D point cloud processing, realizing the effective fusion of multimodal data and solving the problems of existing methods lacking multimodal fusion mechanisms and the single application of deep learning models.

[0028] Furthermore, NURBS surface fitting was performed based on the point cloud of the non-tooling area, and the estimated point cloud of the occluded area was generated by uniform sampling. This enabled accurate completion of the missing parts of the workpiece, avoiding the data loss problem caused by tooling occlusion and ensuring the integrity of the workpiece's 3D model. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a workpiece completion method based on deep learning and surface estimation according to the present invention. Figure 2 This is a schematic diagram of a workpiece completion system based on deep learning and surface estimation according to the present invention. Figure 3 This is a schematic diagram of an electronic device structure according to the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0031] Example 1 like Figure 1 The workpiece completion method shown here, based on deep learning and surface estimation, specifically includes: RGB cameras and LiDAR are used to acquire RGB images of the workpiece and tooling in the working area and LiDAR point clouds; The core principle of this step is multimodal data complementary acquisition. The RGB camera has high-resolution texture and color capture capabilities, which can extract semantic features of the scene and provide a visual foundation for subsequent target segmentation. The LiDAR has accurate three-dimensional spatial ranging capabilities and can acquire point cloud structure information of the scene regardless of lighting conditions, enabling coarse perception of the spatial position of the workpiece and tooling. Through the design of simultaneous acquisition by the two, the technical advantages of each sensor can be fully utilized to make up for the lack of information dimension of a single sensor, providing complete and complementary raw data for subsequent contact area positioning and fine point cloud processing. This is the basic data guarantee for the high accuracy of the entire method.

[0032] Based on the analysis of RGB images and LiDAR point clouds, the contact area between the workpiece and the tooling is determined to achieve coarse positioning. The core logic of this step is to deeply fuse the semantic information of the 2D image with the spatial information of the 3D point cloud, giving the point cloud semantic attributes, thereby achieving rapid and accurate identification of the contact area. The specific process is as follows: A semantic segmentation network is used to segment RGB images into workpieces, fixtures, and backgrounds, obtaining pixel-level semantic masks. These masks can clearly distinguish the pixel ranges of different targets in the image and extract the two-dimensional contour information of the workpieces and fixtures. Through training on a large number of industrial scene images, the semantic segmentation network can accurately learn the texture, shape, and color features of the workpieces, fixtures, and backgrounds, achieving pixel-level target differentiation. Its output semantic mask can clearly define the two-dimensional contour range of the workpieces and fixtures in the image, providing prior semantic labels for subsequent point cloud semantic annotation, avoiding the limitations of traditional point cloud segmentation that relies solely on geometric features. The semantic mask and the LiDAR point cloud are registered to obtain the semantic point cloud. Specifically, the LiDAR point cloud is projected onto the RGB image plane according to the extrinsic calibration parameters between the RGB camera and the LiDAR to obtain the pixel coordinates. By querying the category label of the semantic mask at pixel coordinates, the semantic label is associated and fused with the corresponding LiDAR point cloud to finally obtain the semantic point cloud; The further specific implementation process is as follows: Let a three-dimensional point in the lidar coordinate system be... Based on the extrinsic calibration parameters between the RGB camera and the LiDAR, i.e., the rotation matrix Translation vector Transform this point to the camera coordinate system:

[0033] in, This represents the three-dimensional coordinates of the point in the camera coordinate system. For depth; Using the intrinsic parameter matrix of an RGB camera Including focal length , and the main point , ,Will Project onto the image plane. First, calculate the normalized coordinates:

[0034]

[0035] Then consider camera distortion (such as radial and tangential distortion), for Distortion correction is performed to obtain the corrected normalized coordinates. The distortion model can be the commonly used radial distortion model (parameters). , , ) and tangential distortion model (parameters) , The specific formula is as follows:

[0036]

[0037]

[0038] Finally, the distortion-corrected coordinates are converted into pixel coordinates using the intrinsic parameter matrix:

[0039]

[0040] Received This refers to the pixel coordinates of the 3D point on the RGB image.

[0041] After obtaining pixel coordinates Then, make the following judgment: like and All are within the valid range of the image, and the depth values (Indicating a point in front of the camera), the category label for that pixel location is read from the pre-calculated semantic mask image; these labels include tooling, workpiece, or background. If pixel coordinates exceed the image boundary or depth value If the value is not within the camera's field of view or is located behind the camera, a valid semantic label cannot be obtained, and it can be marked as invalid or ignored.

[0042] Category labels for valid coordinate points Attached to the corresponding lidar point This creates a new attribute field. By iterating through all LiDAR point clouds and repeating the above steps, a semantic label can be assigned to each point, ultimately resulting in a point cloud with semantic information (a semantic point cloud). The data structure of each point in this semantic point cloud can be represented as follows: .

[0043] The extrinsic calibration parameters ensure the spatial registration accuracy between the point cloud and the image, while the pixel coordinate mapping enables the accurate transfer of 2D semantic labels to 3D point clouds, giving the point cloud, which originally only had spatial coordinates, the semantic attributes of "workpiece / tooling / background", thus solving the problem that pure point cloud data is difficult to distinguish targets. Coarse positioning is achieved by detecting the contact area between the workpiece and the tooling based on semantic point cloud; By analyzing the spatial adjacency relationships of point cloud clusters labeled "workpiece" and "tooling" in semantic point clouds, the contact boundary between the two in three-dimensional space can be quickly identified, and the approximate spatial range of the contact area can be delineated. This method breaks through the limitations of traditional pure geometric analysis, using semantic attributes as the core judgment criterion. Even when the geometric features of the contact area are blurred, it can achieve accurate coarse positioning, delineating precise target areas for subsequent high-precision local scanning, avoiding invalid scans, and improving overall processing efficiency.

[0044] Based on the LiDAR point cloud, a global point cloud model of the working area is reconstructed. A reachability analysis is then performed on the global point cloud model to generate the shooting trajectory. The specific steps are as follows: The purpose of stitching, denoising, filtering, and registration operations on the lidar point cloud is to remove noisy and outlier points and reconstruct a complete point cloud model of the working area. This overall point cloud model is used to reconstruct the global 3D structure of the scene; Based on the semantic point cloud, the spatial distribution range of the workpiece is determined from the reconstructed overall point cloud model of the lidar, and the region of interest is delineated accordingly to determine the scanning range that the structured light camera needs to cover, thus avoiding invalid scanning. The region of interest is the overall point cloud model. The workpiece point cloud clusters in the data, and the region of interest can be divided using bounding boxes; The extracted workpiece contour region is subjected to surface normal vector estimation and curvature analysis to generate an initial set of viewpoints for covering the workpiece surface, wherein each viewpoint contains the preset spatial position and optical axis orientation of the structured light camera. First, for each point in the workpiece point cloud cluster, fit a local plane using its neighborhood point set and calculate the normal vector of that point. Note that the normal vector points to the outside of the point cloud surface. Then, the curvature change at each point is calculated. For areas with large curvature (such as edges, holes, and complex surfaces), more viewpoints are needed to cover them; for flat areas with small curvature, the viewpoint density can be appropriately reduced. Finally, a spherical sampling method is adopted, which involves uniformly sampling on a sphere with the workpiece as the center and a radius equal to the optimal working distance. Each sampling point corresponds to a camera position. For each sampled camera position, the vector pointing from that position to the geometric center of the workpiece is calculated as the orientation of the camera's optical axis. For complex curved surfaces, the orientation can be adjusted based on the local normal vector to ensure that the angle between the camera's optical axis and the surface normal vector is within the allowable range, thus generating an initial set of viewpoints. ,in For the camera's spatial position, Orientation of the optical axis; By combining the robot's kinematic model and the obstacle distribution in the workspace, accessibility analysis and collision detection are performed on the initial set of viewpoints to eliminate viewpoints that the robotic arm cannot reach or that pose a collision risk. First, a kinematic model of the first robot (a robot equipped with a structured light camera) is established, including DH parameters (Denavit-Hartenberg parameters), joint constraints, maximum velocity, and acceleration. Then for each viewpoint in the initial viewpoint set The pose is converted into the target pose of the robot's end flange, and inverse kinematics is solved to determine whether there is a set of joint angles that can enable the robot to reach the pose. If there is no solution, or the solved joint angles exceed the joint limits, the viewpoint is discarded. Then, fixed obstacles (such as the second robot, tooling platform, safety fence) and movable obstacles (such as the calibration plate carried by the second robot) in the workspace are constructed as simplified collision models (such as bounding boxes, spheres). For each reachable viewpoint, the straight-line path (or simple interpolation path) of the robot moving from the current position to the target viewpoint is simulated. All intermediate poses on the path are checked for interference with obstacles. If a collision occurs, the viewpoint is eliminated. Finally, the set of usable viewpoints is obtained. These viewpoints satisfy both the robot's kinematic constraints and eliminate the risk of collision. Based on the greedy projection optimization algorithm, the remaining available viewpoints are sorted and redundant points are removed to plan a smooth, continuous and complete structured light camera shooting trajectory. The greedy projection optimization algorithm is as follows: S1. Project the viewpoint coordinates in three-dimensional space onto a two-dimensional plane (or sphere) to reduce the dimensionality of the problem; S2. On the projection plane, a greedy strategy (such as the nearest neighbor method) is used to initially sort the viewpoints: starting from any viewpoint, each time the unvisited viewpoint closest to the current point is selected as the next target; S3. In three-dimensional space, perform local optimization on the initial path. For example, use the 2-opt algorithm: check if there are intersections in the path, and if so, swap the order of nodes to eliminate the intersections and shorten the path.

[0045] S4. Interpolate or fit spline to the turning points in the path to generate a smooth motion trajectory and avoid drastic acceleration and deceleration during robot movement.

[0046] A structured light camera is used to perform a global scan of the work area along the shooting trajectory to acquire a high-density point cloud and reconstruct it, resulting in a fine point cloud model that includes the workpiece and tooling. In particular, the focal length of the structured light camera is set to 300mm-500mm when performing global scanning. Since the structured light camera can achieve an accuracy of 0.1mm, it can capture details that are difficult to detect by traditional methods, such as contact seams and tiny steps. Based on the coarse positioning results, a structured light camera was used to collect local data of the contact area. This local data includes local high-precision depth maps, grayscale images, and local high-density point clouds; The local high-precision depth map records the precise depth information of each pixel, which is the basis for subsequent 3D reconstruction; grayscale images are used to provide high-resolution texture information, reflecting the light and dark changes and edge features of the object's surface; local high-density point clouds are calculated from the depth map, and the point cloud density is much higher than that of LiDAR, which can finely depict the geometric structure of the contact area.

[0047] Fine semantic segmentation is performed on the local data. Based on the semantic segmentation results, the point cloud of the tooling part is removed from the fine point cloud model to obtain the point cloud of the workpiece part. The fine semantic segmentation process includes image and segmentation: local grayscale images are input into a specially trained fine semantic segmentation network, which is optimized for contact areas and can classify pixels more finely, distinguishing between "workpiece side", "tooling side" and "contact seam". Image-point cloud registration: Using the imaging model (intrinsic parameters) of the structured light camera itself and the pose relationship between the camera and the robot end effector, the local semantic mask and the local high-density point cloud are registered at the pixel level. Each point cloud point obtains a corresponding fine semantic label based on its pixel position projected onto the image. Tooling point cloud culling: In a fine point cloud model In the process, based on the fine semantic labels, all point clouds marked as "tooling" are identified and removed, while point clouds marked as "workpiece" are retained. At the same time, for "fuzzy bands" that may exist in the contact area (such as point clouds that are simultaneously projected onto the workpiece and tooling pixels), corrections can be made through confidence thresholds or geometric continuity to ensure the accuracy of segmentation. Finally, the point cloud of the workpiece is obtained. The point cloud at this time has the following characteristics: high precision (from the structured light camera), purity (tooling point cloud has been removed), and complete coverage of the workpiece surface (except for the occluded areas).

[0048] By guiding the structured light camera to acquire local data within its optimal focal length range through coarse positioning, this step captures details that are difficult to detect using traditional methods, such as contact seams and minute steps. This detailed information is not only the basis for accurate segmentation, but also crucial for ensuring that the surface fits the actual workpiece surface during subsequent surface fitting—if these details are missing, the fitted surface may deviate from the actual workpiece, leading to inaccurate completion results.

[0049] The point cloud of the workpiece is predicted and estimated to generate the estimated point cloud of the area occluded by the tooling. The specific steps are as follows: Based on the point cloud of the workpiece, a non-uniform rational B-spline surface is fitted to the workpiece surface to generate a fitted surface. This fitted surface is in the parameter space. The above is performed, and the specific expression is:

[0050] In the formula, As control points, As weight, and They are respectively order and B-spline basis functions of order 1 and This refers to the number of control points in the two parameter directions; The physical meaning of this formula is: points on a curved surface It is a weighted average of all control points; where the control points form a control grid, they attract the surface like "magnets", but the surface does not necessarily pass through them; the weights control the magnitude of the "attraction" of each control point to the surface; the basis functions determine the range of influence of each control point; Furthermore, since the core of fitting is to solve for control points and weights to make the surface as close as possible to the known point cloud, the least squares optimization method is used to set the objective function for fitting, specifically:

[0051] In the formula, The first point cloud of the workpiece A point cloud, for The corresponding parameter values, This represents the total number of point clouds in the workpiece area; At this point, the originally discrete point cloud is "upgraded" into a continuous mathematical model. This surface contains the geometric laws of the workpiece surface—curvature changes, concave and convex features, and edge orientation. Perform uniform sampling based on the fitted surface; In parameter space The calculation is performed internally, and the formula is as follows:

[0052]

[0053] and:

[0054]

[0055] In the formula, and For in parameter space direction and The number of points generated in a direction determines the "detail" of the completed point cloud—the larger the value, the denser the completed point cloud. Substituting the sampling results into the surface equation yields a uniformly distributed estimated point cloud, i.e., each group Substitute the surface equation Calculate the corresponding three-dimensional point coordinates ; These newly generated points are the estimated point cloud of the area occluded by the tooling. They have the following characteristics: Uniform distribution: Because sampling is performed uniformly in the parameter space, the generated point cloud is uniformly distributed on the surface of the curved surface; Geometric consistency: Strictly follows the surface pattern fitted from the known point cloud, and naturally connects with the original surface of the workpiece; Continuous and complete: It covers all the obscured areas without any gaps; splicing to estimate point clouds Point cloud of the workpiece The completed workpiece point cloud is obtained.

[0056] Example 2 like Figure 2 As shown, this embodiment provides a workpiece completion system based on deep learning and surface estimation to implement the workpiece completion method of Embodiment 1 above. The system includes the following components connected in sequence: The multi-source data acquisition module is used to acquire RGB images of the workpiece and tooling and LiDAR point clouds in the working area through an RGB camera and LiDAR. The coarse localization module for the contact area is used for semantic segmentation, point cloud registration and contact area detection based on RGB images and LiDAR point clouds to achieve coarse localization of the contact area between the workpiece and the tooling. The shooting trajectory planning module is used to reconstruct the overall point cloud model of the working area based on the LiDAR point cloud, and generate the shooting trajectory of the structured light camera through accessibility analysis and path optimization. The high-precision scanning module is used to control the structured light camera to perform global and local high-precision scanning along the shooting trajectory to acquire fine point cloud models and local high-precision data. The workpiece point cloud extraction module is used to perform fine semantic segmentation on local data, remove tooling point clouds from the fine point cloud model, and obtain the point cloud of the workpiece part. The occlusion region estimation module is used to perform NURBS surface fitting and uniform sampling on the point cloud of the workpiece part to generate an estimated point cloud of the occluded area by the tooling. The point cloud stitching and completion module is used to stitch and fuse the estimated point cloud with the point cloud of the workpiece to obtain the completed workpiece point cloud.

[0057] The modules work together to achieve fully automated processing from multi-source data acquisition to complete workpiece point cloud completion, improving the accuracy and efficiency of workpiece completion. Example 3 like Figure 3 An electronic device is shown, comprising: a processor, a memory, and a communication interface; The memory is used to store the executable instructions of the processor; The processor is configured to execute the workpiece completion method based on deep learning and surface estimation described in Embodiment 1 above by executing the executable instructions.

[0058] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the workpiece completion method based on deep learning and surface estimation described in Embodiment 1 above.

[0059] The aforementioned electronic device and readable storage medium solidify the workpiece completion method of the present invention into an executable program, which can be quickly deployed and run in industrial automated testing equipment, realize the engineering application of workpiece completion, and is applicable to various industrial fields such as aerospace, precision manufacturing, and rail transportation that require complete reconstruction of workpiece three-dimensional models; The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A workpiece completion method based on deep learning and surface estimation, characterized in that, Specifically, it includes: RGB cameras and LiDAR are used to acquire RGB images of the workpiece and tooling in the working area and LiDAR point clouds; Based on the analysis of RGB images and LiDAR point clouds, the contact area between the workpiece and the tooling is determined to achieve coarse positioning; Based on the lidar point cloud, a global point cloud model of the working area is reconstructed, and an accessibility analysis is performed on the global point cloud model to generate the shooting trajectory. A structured light camera is used to perform a global scan of the working area along the shooting trajectory to obtain a high-density point cloud and reconstruct it, resulting in a fine point cloud model that includes the workpiece and tooling. Based on the coarse positioning results, a structured light camera was used to collect local data of the contact area. Fine semantic segmentation is performed on local data, and the point cloud of the tooling part is removed from the fine point cloud model to obtain the point cloud of the workpiece part. Perform predictive estimation analysis on the point cloud of the workpiece to generate an estimated point cloud of the area occluded by the tooling. By stitching together the estimated point cloud and the point cloud of the workpiece portion, the completed workpiece point cloud is obtained.

2. The workpiece completion method based on deep learning and surface estimation according to claim 1, characterized in that: The analysis based on RGB images and LiDAR point clouds determines the contact area between the workpiece and the tooling, achieving coarse positioning. The specific steps are as follows: A semantic segmentation network is used to segment RGB images into workpieces, fixtures, and backgrounds to obtain pixel-level semantic masks. The semantic mask and the lidar point cloud are registered to obtain the semantic point cloud. Coarse positioning is achieved by detecting the contact area between the workpiece and the tooling based on semantic point cloud.

3. The workpiece completion method based on deep learning and surface estimation according to claim 2, characterized in that: The process of registering the semantic mask and the LiDAR point cloud to obtain the semantic point cloud specifically involves: projecting the LiDAR point cloud onto the RGB image plane based on the extrinsic calibration parameters between the RGB camera and the LiDAR to obtain the pixel coordinates. By querying the category label of the semantic mask at pixel coordinates, the semantic label is associated and fused with the corresponding LiDAR point cloud to finally obtain the semantic point cloud.

4. The workpiece completion method based on deep learning and surface estimation according to claim 1, characterized in that: The overall point cloud model of the working area is reconstructed based on the LiDAR point cloud. A reachability analysis is performed on the overall point cloud model to generate the shooting trajectory. The specific steps are as follows: The lidar point cloud is stitched, denoised, filtered, and registered to reconstruct the overall point cloud model of the working area. Based on the semantic point cloud, the spatial distribution range of the workpiece is determined from the reconstructed overall point cloud model of the lidar, and the region of interest is delineated accordingly to determine the scanning range that the structured light camera needs to cover. The extracted workpiece contour region is subjected to surface normal vector estimation and curvature analysis to generate an initial set of viewpoints for covering the workpiece surface, wherein each viewpoint contains the preset spatial position and optical axis orientation of the structured light camera. By combining the robot's kinematic model and the obstacle distribution in the workspace, accessibility analysis and collision detection are performed on the initial set of viewpoints to eliminate viewpoints that the robotic arm cannot reach or that pose a collision risk. Based on the greedy projection optimization algorithm, the remaining available viewpoints are sorted and redundant points are removed to plan a smooth, continuous and complete structured light camera shooting trajectory.

5. The workpiece completion method based on deep learning and surface estimation according to claim 1, characterized in that: When collecting local data of the contact area using a structured light camera based on the coarse positioning results, the focal length of the structured light camera is set to 300mm-500mm.

6. The workpiece completion method based on deep learning and surface estimation according to claim 1, characterized in that: The local data includes local high-precision depth maps, grayscale images, and local high-density point clouds.

7. The workpiece completion method based on deep learning and surface estimation according to claim 1, characterized in that: The specific steps for predicting and estimating the point cloud of the workpiece portion to generate an estimated point cloud of the area occluded by the tooling are as follows: Based on the point cloud of the workpiece, a non-uniform rational B-spline surface is fitted to the workpiece surface to generate the fitted surface. Perform uniform sampling based on the fitted surface; Substituting the sampling results into the surface equation yields a uniformly distributed estimated point cloud.

8. The workpiece completion method based on deep learning and surface estimation according to claim 7, characterized in that: The non-uniform rational B-spline surface fitting in parameter space The above is performed, and the specific expression is: In the formula, As control points, As weight, and They are respectively order and B-spline basis functions of order 1 and This represents the number of control points in the two parameter directions.

9. The workpiece completion method based on deep learning and surface estimation according to claim 8, characterized in that: The objective function for fitting the non-uniform rational B-spline surface is: In the formula, The first point cloud of the workpiece A point cloud, for The corresponding parameter values, This represents the total number of point clouds in the workpiece section.

10. The workpiece completion method based on deep learning and surface estimation according to claim 9, characterized in that: The process of uniform sampling based on the fitted surface is specifically performed in the parameter space. The calculation is performed internally, and the formula is as follows: and: In the formula, and For in parameter space direction and The number of points generated in the direction.

Images