A fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing
By constructing a red closed boundary around the area to be polished, and using a depth camera and color space feature fusion segmentation model, a three-dimensional point cloud can be quickly extracted in the two-dimensional image domain. This solves the problem of rapid and accurate acquisition of the area to be polished in robotic polishing, simplifies the system structure, and reduces costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CENT SOUTH UNIV
- Filing Date
- 2026-04-02
- Publication Date
- 2026-06-09
Smart Images
Figure CN122175953A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of robot vision perception, image processing, 3D point cloud segmentation, and robot polishing technology, specifically to a fast point cloud segmentation method based on image mapping for local workpiece polishing. This method constructs a red closed boundary around the area to be polished, achieving rapid target region localization in a 2D image domain. Then, using an ordered point cloud that corresponds one-to-one with the pixels of the color image, the 2D target region is accurately mapped to 3D space, enabling rapid extraction of the point cloud of the area to be polished. Background Technology
[0002] In robotic grinding and polishing processes, typically only a localized area of the workpiece surface needs to be processed. Therefore, quickly and accurately acquiring the 3D point cloud corresponding to the area to be ground and polished is a crucial prerequisite for subsequent path planning, attitude adjustment, and processing control. This is especially true for large, complex curved workpieces, where the processing area is usually located on a continuous, smooth surface with a limited local area and complex on-site conditions. Failure to efficiently obtain the point cloud of the target area will directly impact the automation level and processing quality of the robotic localized grinding and polishing task.
[0003] In existing technologies, point cloud acquisition of the processing area typically involves two technical approaches. The first approach is to directly segment the region in the 3D point cloud domain, using methods such as clustering, normal vector analysis, curvature constraints, region growing, model fitting, or deep learning instance segmentation to extract the target region from the original point cloud. Patent CN116630623B discloses a workpiece point cloud instance segmentation method for industrial scenarios. This method can improve the point cloud segmentation accuracy using deep learning and is applicable to workpiece recognition and segmentation tasks in industrial scenarios. However, its technical approach relies on point cloud annotation, model training, and inference processes, resulting in high overall computational complexity and strong dependence on computing power and datasets. For robot local grinding and polishing tasks that require rapid interaction and real-time extraction of local processing area point clouds, its segmentation efficiency, deployment cost, and specificity still have certain limitations.
[0004] The second type of technical approach involves first determining the target region in the two-dimensional image domain, and then mapping the image region onto a three-dimensional point cloud. This type of method leverages the advantages of images in terms of color, texture, and edge representation, which can reduce the computational complexity of pure three-dimensional processing to some extent and improve interaction efficiency. Patent CN119131061B discloses a method for segmenting industrial parts images under uneven illumination. It traverses the pixels of the workpiece image and maps them to the RGB color space. This method proposes a solution to the two-dimensional segmentation problem of industrial parts images under conditions of uneven illumination and local reflection, and can improve image segmentation accuracy. However, its processing object is essentially still a two-dimensional image, and it only obtains the image segmentation result of the part region, and cannot directly obtain local three-dimensional point cloud data for robot grinding and polishing path planning.
[0005] Therefore, there is an urgent need for a method that can stably determine closed boundaries in a two-dimensional image domain and utilize the correspondence between image pixels and point clouds to achieve rapid and accurate extraction of point clouds in the area to be polished, so as to meet the needs of robot local polishing tasks for rapid interaction, accurate segmentation of target areas, and subsequent path planning input. Summary of the Invention
[0006] This invention establishes a red closed boundary around the area to be polished. A single depth camera mounted on a robotic arm simultaneously acquires RGB color image data and depth data of the workpiece from the same perspective. Based on camera calibration parameters, the depth map is aligned to the color image coordinate system, and an ordered point cloud with the same resolution and pixel grid as the color image, corresponding one-to-one with the image pixel positions, is constructed. In the two-dimensional image domain, a multi-color space feature fusion segmentation model based on HSV color space, RGB channel difference, and Lab color space responses is used to detect the red closed boundary and generate a binary mask of the red boundary. Then, through morphological closure repair, closed contour filtering, and connected component area constraints, the final mask of the area to be polished is obtained. Finally, based on the correspondence between the target pixel position in the final mask and the midpoint position of the ordered point cloud, the three-dimensional point cloud data corresponding to the target pixel is extracted to obtain the point cloud of the area to be polished. This method achieves rapid mapping and extraction from a two-dimensional image region to a three-dimensional point cloud region without the need for additional industrial cameras, line laser sensors, or other multi-source sensors.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] A fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing includes the following steps:
[0009] Step S1: Construct a red closed boundary around the target area on the surface of the workpiece to be ground and polished;
[0010] Step S2: Using a robotic arm equipped with a depth camera, simultaneously acquire RGB color image data and depth data of the workpiece to be ground and polished from the same perspective. Based on the camera calibration parameters, align the depth map to the color image coordinate system to construct an ordered point cloud with the same resolution and pixel grid as the color image and corresponding one-to-one with the image pixel position.
[0011] Step S3: In the two-dimensional image domain, red closed boundary detection is performed on the acquired RGB color image. The red closed boundary detection adopts a multi-color space feature fusion segmentation model based on HSV color space, RGB channel difference and Lab color space response, and outputs a red boundary binary mask.
[0012] Step S4: Morphological closing operation is performed on the red boundary binary mask to repair it, so as to enhance the boundary connectivity and suppress debris noise; then the outer contour is extracted, the closed contour with the largest area is selected, and the internal region of the closed boundary is filled. Then, combined with the area constraint of the connected domain, the final mask of the area to be polished is obtained.
[0013] Step S5: Based on the correspondence between the target pixel position in the final mask of the area to be polished and the position of the midpoint in the ordered point cloud, extract the three-dimensional point cloud data corresponding to the target pixel to obtain the point cloud of the area to be polished.
[0014] Furthermore, in step S2, after aligning the depth map to the color image coordinate system, the pixel coordinates of the aligned depth map and color image are converted into three-dimensional point cloud coordinates based on the camera intrinsic parameter matrix to construct an ordered point cloud; the ordered point cloud maintains the same row and column arrangement as the color image so that a direct correspondence is established between the image pixel position and the point cloud point position.
[0015] Furthermore, in step S3, the red hue range is initially screened based on the HSV color space, and an initial red candidate region is constructed by combining saturation and brightness dynamic thresholds; in the RGB channel space, a red dominance criterion is constructed by the difference between the red channel and the green and blue channels to suppress regions with similar overall hues but insufficient red dominance; in the Lab color space, an adaptive threshold response mask is constructed by utilizing the high response of the a channel to red targets to enhance the detection effect of red closed boundaries.
[0016] Furthermore, in step S4, the area of the repaired closed boundary contour is screened, and the target contour with the largest area is selected first as the boundary of the area to be polished; on this basis, the area inside the contour is filled, and the final mask of the area to be polished is obtained through the area constraint of the connected domain.
[0017] Furthermore, in step S5, the target pixels in the final mask of the area to be polished are used to retrieve the corresponding three-dimensional points in the ordered point cloud according to the row and column positions consistent with the color image, so as to complete the point cloud extraction of the area to be polished.
[0018] Beneficial effects
[0019] The beneficial effects of this invention are as follows:
[0020] 1. This invention constructs a red closed boundary around the area to be polished on the workpiece, transforming the local processing area that was originally difficult to be stably separated directly from the workpiece point cloud into a target area that can be quickly identified in the two-dimensional image domain, thus significantly reducing the computational complexity caused by direct segmentation of the three-dimensional point cloud.
[0021] 2. This invention only requires a single depth camera to simultaneously acquire RGB color image data and depth data, without the need to add industrial cameras, line laser sensors or other heterogeneous sensors, to complete the detection of two-dimensional target areas and the extraction of three-dimensional point cloud areas. Therefore, the system structure is simpler, the deployment cost is lower, and the calibration process is more simplified.
[0022] 3. This invention adopts a red boundary detection method that integrates features from multiple color spaces, including HSV, RGB, and Lab. This method can constrain and enhance the red boundary from different color expression dimensions. Compared with a single color space threshold segmentation method, it has better boundary detection stability and anti-interference ability.
[0023] 4. The point cloud of the area to be polished obtained by this invention can be directly used as input data for robot polishing planning and trajectory generation, providing a stable, reliable and efficient data foundation for automatic polishing of local workpieces. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;
[0025] Figure 2 A schematic diagram of a robotic arm equipped with a depth camera to synchronously acquire RGB color image data;
[0026] Figure 3 A schematic diagram for constructing an ordered point cloud of the same resolution;
[0027] Figure 4 This is a schematic diagram of red boundary detection based on the fusion of features from multiple color spaces including HSV, RGB, and Lab.
[0028] Figure 5 A schematic diagram of morphological restoration and maximum contour filtering for the red boundary mask;
[0029] Figure 6 This is a schematic diagram showing the final mask generation for the area to be polished.
[0030] Figure 7 A schematic diagram of mapping an image mask onto an ordered point cloud and extracting the point cloud of the area to be polished. Detailed Implementation
[0031] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.
[0032] like Figure 1 As shown, this embodiment provides a fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing, which mainly includes the following five steps:
[0033] Step S1: Construct a red closed boundary around the target area on the surface of the workpiece to be polished; in one embodiment, the red closed boundary can be formed by attaching red marking tape, red strip, or red patch material, or by spraying red marking pigment, red ink, or a peelable red marking coating onto the workpiece surface.
[0034] Step S2: The robotic arm, equipped with a depth camera, simultaneously acquires RGB color image data and depth data of the workpiece to be ground and polished from the same perspective; both are from the same source. Each point in the ordered point cloud corresponds one-to-one with the image pixel position.
[0035] Alignment correction of the depth map is performed using camera calibration parameters to generate an aligned depth map with the same resolution and pixel grid as the color image. Alignment depth map With color illustration Share the same pixel coordinate system ,in , The pixel coordinates of the aligned depth map and color map are converted into 3D point cloud coordinates to construct an ordered point cloud:
[0036]
[0037] The formula for transforming 3D point cloud coordinates is:
[0038]
[0039] In the formula, For the camera intrinsic parameter matrix, , These are the focal lengths of the camera in the x and y directions, respectively. , These are the camera principal point coordinates and pixel indexes. Strictly isomorphic to the point cloud index, the ordered point cloud has a one-to-one correspondence with the image pixels, thus constructing an ordered point cloud.
[0040] Step S3: In the two-dimensional image domain, a multi-color space feature fusion segmentation model is constructed based on the HSV color space, RGB channel difference, and Lab color space response to detect red closed boundaries and output a binary mask of the red boundaries. Specific sub-steps are as follows:
[0041] Step S3.1: Initial red candidate region extraction based on HSV color space. The RGB color image is converted to HSV color space, and the red hue is initially screened using the hue component H. Based on the distribution characteristics of red on the HSV color wheel, the program selects the intervals [0,10] and [170,180] as the red candidate ranges respectively.
[0042]
[0043] in, Represents pixels Does it satisfy the red hue constraint?
[0044] Dynamic threshold constraints for saturation and brightness are also introduced in the HSV color space. Let the overall average saturation and average brightness of the image be respectively... and The dynamic threshold is then defined as:
[0045]
[0046]
[0047] in, The threshold value is the saturation level. This is the lower threshold for brightness.
[0048] This allows for the construction of saturation and brightness constraint masks:
[0049]
[0050]
[0051] This leads to the initial red candidate regions based on HSV:
[0052]
[0053] The above processing achieved the initial location of the candidate region for the red boundary.
[0054] Step S3.2: Construct the red dominance criterion in the RGB channel space based on the RGB channel difference. Let the red, green, and blue components of a pixel in the preprocessed image be R(u,v), G(u,v), and B(u,v), respectively, then define:
[0055]
[0056]
[0057] When the red channel has a sufficient advantage over both the green and blue channels, the pixel can be considered to meet the red-dominant condition, that is:
[0058]
[0059] This criterion directly utilizes the difference relationship between the three color channels to further suppress areas with similar overall hues but insufficient red dominance.
[0060] Step S3.3: Red response enhancement based on Lab color space. The preprocessed image is converted to the Lab color space, and the α channel is extracted for threshold response enhancement. Since the α channel describes the color change in the red-green direction, it has a high response to red targets. Let the mean of the α channel of the image be... Then the following adaptive threshold is used:
[0061]
[0062] Then, a Lab red response mask is constructed:
[0063] Define comprehensive color support items:
[0064] While satisfying the basic HSV constraints, more stringent red detection results are obtained:
[0065] The final red line detection mask is defined as follows:
[0066]
[0067] This yields the refined extraction results of the red boundary. That is, a red closed boundary mask, such as Figure 4 As shown.
[0068] Step S4: Morphological restoration of the red boundary mask and generation of the mask for the area to be polished. Specific sub-steps are as follows:
[0069] Step S4.1: Introduce morphological operations to the red boundary mask to perform geometric constraints and noise suppression on the threshold segmentation results. The morphological closing operation formula is:
[0070]
[0071] Where K is a structuring element. For expansion operation, This is a corrosion operation.
[0072] After closing the operation, boundary connectivity can be enhanced and fragment noise can be suppressed, such as... Figure 5 As shown.
[0073] Step S4.2: Maximum Contour Filtering. Extract the outer contours from the repaired boundary mask, calculate the area of each contour, and filter out the contour with the largest area:
[0074]
[0075] in The area of the i-th contour is calculated using Green's formula:
[0076]
[0077] The closed boundary contour after repair is extracted, and the internal region of the contour is filled using a region filling algorithm to obtain a mask for the area to be polished. ;
[0078] Step S4.3: Perform connected component analysis on the closed boundary contour and / or filled region, setting a preset area threshold as follows. Calculate the area of each connected component. Select the one that satisfies The target connected region is used as the connected region corresponding to the area to be polished; morphological constraints are applied to the mask corresponding to the target connected region to obtain a simply connected or a final mask for the area to be polished that meets preset connectivity requirements. The pixel values of the mask are defined as follows:
[0079]
[0080] This mask characterizes the target area to be polished within a closed boundary, such as... Figure 6 As shown.
[0081] Step S5: Point cloud extraction of the area to be polished based on image mapping. Image pixel coordinates. The point cloud index idx satisfies:
[0082]
[0083] Where W is the image width (i.e., the width of the ordered point cloud). For pixel column coordinates, Here, H represents the pixel row coordinates, and H represents the image height.
[0084] Combined with the obtained mask of the area to be polished ,when When, take the corresponding pixel coordinates Mapped point cloud All extracted point clouds constitute the point cloud of the area to be ground and polished. Complete the extraction of point cloud data for the area to be ground and polished, such as... Figure 7 As shown in the diagram, the above mapping relationship enables rapid segmentation and extraction of point clouds from the target region of a two-dimensional image to the point cloud of the region to be polished in three dimensions.
[0085] In one specific embodiment, a red closed boundary is first constructed around the target area on the surface of the workpiece to be polished. Then, a robotic arm equipped with a depth camera simultaneously acquires RGB color image data and depth data. An ordered point cloud of the same resolution is constructed by aligning the depth map with the color map and combining it with the camera intrinsic parameter matrix. Then, a red boundary mask is extracted in the two-dimensional image domain based on the fusion of HSV, RGB and Lab multi-color space features. After morphological repair, maximum contour filtering, region filling and connected component constraints, the final mask of the area to be polished is obtained. Finally, the point cloud of the area to be polished is extracted according to the correspondence between the target pixel position in the mask of the area to be polished and the midpoint position of the ordered point cloud, which serves as the input for the subsequent polishing planning of the robotic arm.
[0086] Examples of the present invention have been described above with reference to the accompanying drawings. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be defined by the claims.
Claims
1. A fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing, characterized in that, Includes the following steps: S1: Construct a red closed boundary around the target area on the surface of the workpiece to be ground and polished; S2: Using a robotic arm equipped with a depth camera, RGB color image data and depth data of the workpiece to be ground and polished are simultaneously acquired from the same perspective. Based on the camera calibration parameters, the depth map is aligned to the color image coordinate system to construct an ordered point cloud with the same resolution and pixel grid as the color image and corresponding one-to-one with the pixel position of the image. S3: In the two-dimensional image domain, red closed boundary detection is performed on the acquired RGB color image. The red closed boundary detection adopts a multi-color space feature fusion segmentation model based on HSV color space, RGB channel difference and Lab color space response, and outputs a red boundary binary mask. S4: Perform morphological closing operation repair on the red boundary binary mask to enhance boundary connectivity and suppress fragment noise; Then, the outer contour is extracted, the closed contour with the largest area is selected, and the internal area of the closed boundary is filled. Finally, the area constraint of the connected domain is combined to obtain the final mask of the area to be polished. S5: Based on the correspondence between the target pixel position in the final mask of the area to be polished and the position of the midpoint in the ordered point cloud, extract the three-dimensional point cloud data corresponding to the target pixel to obtain the point cloud of the area to be polished.
2. The fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing according to claim 1, characterized in that, In step S2, after aligning the depth map to the color image coordinate system, the pixel coordinates of the aligned depth map and color image are converted into three-dimensional point cloud coordinates based on the camera intrinsic parameter matrix to construct an ordered point cloud. The ordered point cloud maintains the same row and column arrangement as the color image so that a direct correspondence is established between the image pixel position and the point cloud point position.
3. The fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing according to claim 1, characterized in that, In step S3, the red hue range is initially screened based on the HSV color space, and an initial red candidate region is constructed by combining saturation and brightness dynamic thresholds. In the RGB channel space, a red dominance criterion is constructed by the difference between the red channel and the green and blue channels to suppress regions with similar overall hues but insufficient red dominance. In the Lab color space, an adaptive threshold response mask is constructed by utilizing the high response of the a channel to red targets to enhance the detection effect of red closed boundaries.
4. The fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing according to claim 1, characterized in that, In step S4, the area of the repaired closed boundary contour is screened, and the target contour with the largest area is selected as the boundary of the area to be polished. On this basis, the area inside the contour is filled, and the final mask of the area to be polished is obtained by the area constraint of the connected domain.
5. The fast point cloud segmentation method based on image mapping for local workpiece grinding and polishing according to claim 1, characterized in that, In step S5, the target pixels in the final mask of the area to be polished are used to retrieve the corresponding three-dimensional points in the ordered point cloud according to the row and column positions consistent with the color image, so as to complete the point cloud extraction of the area to be polished.