T-shaped shaft support base part point cloud segmentation method based on image guidance

Abstract

The invention discloses a point cloud segmentation method for a T-shaped shaft support base part based on image guidance, and belongs to the technical field of robot industrial application, and the method comprises the steps: S1, making a T-shaped shaft support base part data set: collecting a T-shaped shaft support base part image, and carrying out the preprocessing of the image to improve the sample number of the data set and the quality of the image; s2, constructing a distinguishing feature attention module: inputting the feature map X of the T-shaped shaft bracket base part image into a three-branch structure, and outputting the feature map; s3, constructing an improved YOLOV8 network: adding the discriminant feature attention module constructed in the step S2 between a trunk layer and a prediction layer in the YOLOV8 network, and adding a PointSeg module behind the prediction layer; and S4, retaining the corresponding point cloud in the segmentation mask. According to the method, the problems of low point cloud segmentation speed and low precision of the T-shaped shaft bracket base part can be solved.

Description

Technical Field

[0001] The invention relates to the technical field of industrial application of robots, and in particular to a point cloud segmentation method of a T-shaped shaft bracket base part based on image guidance. Background Art

[0002] The grasping of industrial parts of T-axis bracket base is one of the very important contents in the field of industrial application of robots. It can improve industrial production efficiency and reduce labor costs to a certain extent. Industrial parts of T-axis bracket base are usually placed in a material frame, and the parts are randomly stacked. The material frame is a deep frame. Such placement conditions cause great difficulties for the identification of parts. Therefore, how to perform point cloud segmentation on the T-axis bracket base parts stacked in a deep frame in a complex scene is of great significance to the subsequent point cloud alignment and grasping tasks. The point cloud segmentation of T-axis bracket base parts is the data basis for subsequent tasks.

[0003] There are currently three main methods for point cloud segmentation of industrial parts of T-axis bracket base. The first is to perform manual feature extraction through traditional point cloud algorithms, and then perform point cloud segmentation through point cloud clustering. This method can segment point clouds to a certain extent, but for point clouds with deep frame stacking, it is impossible to segment individual part point clouds, and there will be many noisy point clouds. At the same time, the manual feature extraction method is also suitable for complex industrial scenes.

[0004] The second is to directly process the point cloud through deep learning: PointNet pioneered the direct processing of point cloud data for classification and segmentation tasks. In order to solve the problem of disorder of point cloud data, symmetric functions are used to process high-dimensional features after multi-layer perceptron; but because its input data is independent points, the connection between points is ignored and the ability to capture local feature information between points is lacking. In the prior art, a new convolution method is also used, which designs a point convolution structure in which each point is used as the center to operate other point clouds in the convolution kernel area. The above two methods are to directly process point clouds for point cloud segmentation, which can deal with complex scenes for point cloud segmentation, but direct point cloud processing takes a long time and cannot achieve the effect of real-time detection in industry. At the same time, the data set of industrial parts of T-shaped shaft bracket base for point cloud segmentation required for training the network is difficult to obtain, and the cycle of making data is long. When T-shaped shaft bracket base parts of different specifications appear, model optimization is difficult.

[0005] The third method is to process the image by image projection and then perform point cloud segmentation. This method, such as SqueezeSeg and RangeNet++, can improve the speed of point cloud segmentation, but the detection effect of the T-axis bracket base parts is not good, and it is easy to have false omissions, and it is impossible to achieve a good balance between detection speed and detection performance.

[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0007] The object of the present invention is to provide a point cloud segmentation method for a T-axis bracket base part based on image guidance, which can solve the problems of slow speed and low precision of point cloud segmentation of the T-axis bracket base part.

[0008] To achieve the above object, the present invention provides a point cloud segmentation method for a T-shaped shaft bracket base part based on image guidance, comprising the following steps:

[0009] S1: Create a T-shaped shaft bracket base part dataset: collect T-shaped shaft bracket base part images and preprocess the images to improve the number of samples in the dataset and the quality of the images;

[0010] S2: Construct a discriminative feature attention module: input the feature map X of the T-shaped shaft bracket base part image into the three-branch structure and output the feature map;

[0011] S3: Build an improved YOLOV8 network: Add the discriminative feature attention module constructed in step S2 between the backbone layer and the prediction layer in the YOLOV8 network, and add a PointSeg module after the prediction layer; input the output result of the prediction layer and the point cloud data into the PointSeg module to obtain the coordinates of the three-dimensional point cloud projected onto the plane;

[0012] S4: The coordinates of the 3D point cloud projected onto the plane are compared and screened with the points in the segmentation mask, and the corresponding point cloud in the segmentation mask is retained.

[0013] In one embodiment of the present invention, in step S1, the step of preprocessing the image includes: flipping the image horizontally and vertically; and adjusting the brightness of the image.

[0014] In one embodiment of the present invention, the adjusting the brightness of the picture includes: increasing the brightness of the picture by 10-20% or decreasing the brightness of the picture by 10-20%.

[0015] In one embodiment of the present invention, the adjusting the brightness of the image includes: increasing or decreasing the brightness of the image by 15%.

[0016] In one embodiment of the present invention, step S2 includes the following steps:

[0017] S201: Input the feature map X input by the first branch in the three-branch structure into the posture attention module, and output the first branch feature map X1; wherein the feature map X refers to a set of feature vectors generated by convolution after the image of the T-shaped shaft bracket base part is input into the network;

[0018] S202: performing channel multiplication on the feature map X input by the second branch and the feature map X1 of the first branch, and outputting the feature map X2 of the second branch;

[0019] S203: Input the second branch feature map X2 into the channel attention module, and output the attention feature map X3;

[0020] S204: Multiply the feature map X inputted by the third branch by the attention feature map X3 to obtain the third branch feature map X4, which is the output result of the feature attention mechanism.

[0021] In one embodiment of the present invention, in step S201, the posture attention module normalizes the feature map X to obtain the horizontal feature X W And the vertical feature X H , as shown in the following formula:

[0022] X H =Y Avgpool (BatchNorm(Relu(X)))

[0023] X W =X Avgpool (BatchNorm(Relu(X)))

[0024] Among them, Y Avgpool is the 1*1 average pooling in the vertical direction, X Avgpool It is the 1*1 average pooling in the horizontal direction, Relu is the activation function, and BatchNorm is the normalization operation;

[0025] The first branch feature map X1 output includes the horizontal feature X W And the vertical feature X H , as shown in the following formula:

[0026] X1=X H +X W .

[0027] In one embodiment of the present invention, in step S203, in the channel attention module, a position attention mechanism operation is performed on the second branch feature map X2, as shown in the following formula:

[0028] X3=MC(X2)

[0029] Among them, MC is the operation of the position attention mechanism.

[0030] In one embodiment of the present invention, the improved YOLOV8 segmentation network further includes: modifying the prediction layers of small, medium and large scales output by the prediction layer into medium target and large target prediction layers.

[0031] In one embodiment of the present invention, in step S3, the specific process of the PointSeg layer projecting the three-dimensional point cloud to the plane coordinates is as follows:

[0032] S301: Get the intrinsic parameter matrix K of the current camera:

[0033]

[0034] Among them, f x and f y are the focal lengths of the camera in the x and y directions respectively; c x and c y are the coordinates of the center of the image;

[0035] S302: Convert the coordinates (X, Y, Z) of the three-dimensional point in the camera coordinate system to normalized image plane coordinates:

[0036]

[0037] And use the intrinsic parameter matrix K of the current camera to convert the normalized image plane coordinates to pixel coordinates (u, v)

[0038] u=f x ×x'+c x

[0039] v=f y ×y'+c y .

[0040] In one embodiment of the present invention, in step S1, the samples in the data set are divided into a training set and a validation set at a ratio of 9:1, which are used to train and validate the model established in the subsequent step S3.

[0041] Compared with the prior art, the image-guided point cloud segmentation method for the base part of a T-axis bracket according to the present invention is based on the yolov8 segmentation network, and a discriminative feature attention mechanism is added to focus the network's attention on the discriminative features of the parts; at the same time, according to the actual specifications of the base part of the T-axis bracket, the small target prediction layer is reduced, the medium target and large target prediction layers are expanded, and the part segmentation accuracy is improved; finally, a PointSeg point cloud segmentation module is introduced to perform projection segmentation on the point cloud information. Therefore, the present invention achieves a good balance between the accuracy and speed of the point cloud segmentation of the base part of the T-axis bracket, improves the efficiency of the point cloud segmentation of the base part of the T-axis bracket, and improves the rhythm of the industrial production line. The method of the present invention is fast and accurate, and is more suitable for deployment in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a point cloud segmentation method of a T-axis bracket base part based on image guidance according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of the structure of an attention mechanism according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of the structure of an improved YOLOV8 network according to an embodiment of the present invention;

[0045] Figure 4 is a picture segmentation effect diagram after step S3 according to an embodiment of the present invention;

[0046] Figure 5 This is a point cloud segmentation effect diagram after step S4 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0048] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0049] like Figures 1 to 3 As shown, according to a preferred embodiment of the present invention, a point cloud segmentation method for a T-shaped shaft bracket base part based on image guidance comprises the following steps:

[0050] S1: Making a T-shaped shaft bracket base part dataset: Collect T-shaped shaft bracket base part images and preprocess the images to improve the number of samples in the dataset and the quality of the images.

[0051] The image obtained after the camera takes a picture of the T-axis bracket base part will automatically obtain a point cloud image and a 2D image.

[0052] The step of preprocessing the image includes:

[0053] 1. Flip the image horizontally and vertically to increase the number of samples in the data set, thereby increasing the diversity of samples so that the subsequently trained model can adapt to various situations.

[0054] 2. Adjust the brightness of the image to reduce the effect of lighting conditions on the imaging of reflective objects due to the high reflective properties of industrial parts. Specifically, increase or decrease the brightness of the image by 10-20%, preferably increase or decrease the brightness of the image by 15%. Since the captured image is a binary image, this operation can enhance the contrast of the image.

[0055] The samples in the data set are divided into a training set and a validation set at a ratio of 9:1, which are used to train and validate the model established in the subsequent step S3 to improve the accuracy.

[0056] S2: Construct a discriminative feature attention module: Input the feature map X of the T-shaped shaft bracket base part image into the three-branch structure and output the feature map.

[0057] The three-branch structure aims to perform multi-faceted analysis and enhancement on the input feature map through different processing paths to extract more discriminative and effective features.

[0058] Specifically, Figure 2 As shown in the schematic diagram of the attention mechanism, step S2 includes the following steps:

[0059] S201: Input the feature map X of the first branch input in the three-branch structure into the posture attention module, and output the first branch feature map X1. The feature map X refers to a set of feature vectors generated by convolution after the image of the T-shaped shaft bracket base part is input into the network, and each element thereof refers to the vector value after convolution. After several layers of the network, the feature vector map formed before the discriminant feature attention mechanism is obtained.

[0060] In the pose attention module, the feature map X is normalized to obtain the horizontal feature X W And the vertical feature X H , as shown in the following formula:

[0061] X H =Y Avgpool (BatchNorm(Relu(X)))

[0062] X W =X Avgpool (BatchNorm(Relu(X)))

[0063] Among them, Y Avgpool is the 1*1 average pooling in the vertical direction, X Avgpool It is the 1*1 average pooling in the horizontal direction, Relu is the activation function, and BatchNorm is the normalization operation.

[0064] The first branch feature map X1 output includes the horizontal feature X W And the vertical feature X H , as shown in the following formula:

[0065] X1=X H +X W .

[0066] In the above formula, after the image passes through the network, a vector of dimensions (B, C, W, H) is generated. The Y direction is the operation on the H dimension, that is, the height direction. The average pooling operation is to average the data in a window.

[0067] S202: Perform channel multiplication on the feature map X input by the second branch and the feature map X1 of the first branch, and output the feature map X2 of the second branch.

[0068] S203: Input the second branch feature map X2 into the channel attention module and output the attention feature map X3.

[0069] In the channel attention module, the position attention mechanism is operated on the second branch feature map X2, as shown in the following formula:

[0070] X3=MC(X2)

[0071] Among them, MC is the operation of the position attention mechanism, which is a common and common operation mode.

[0072] S204: Multiply the feature map X inputted by the third branch by the attention feature map X3 to obtain the third branch feature map X4, which is the output result of the feature attention mechanism.

[0073] S3: Build an improved YOLOV8 network: Add the discriminative feature attention module constructed in step S2 between the backbone layer and the prediction layer in the YOLOV8 network, and add a PointSeg module after the prediction layer; input the output result of the prediction layer and the point cloud data into the PointSeg module to obtain the coordinates of the three-dimensional point cloud projected to the plane.

[0074] like Figure 3As shown, it is a schematic diagram of the structure of the improved YOLOV8 network, showing the location of the discriminative feature attention module and the PointSeg module. In this step S3, the PointSeg module is added after the output result to project the three-dimensional point cloud to the plane coordinates; wherein, the input of the PointSeg module is the segmentation mask and the original point cloud data after the YOLOV8 segmentation network inference, and the original point cloud data can be directly obtained through the point cloud image in the image of the T-shaped shaft bracket base part photographed in step S1.

[0075] The original YOLOV8 segmentation network consists of three parts: feature extraction layer, backbone layer and prediction layer. In this step S3, the YOLOV8 segmentation network is also improved as follows: the prediction layers of small, medium and large scales output by the prediction layer are modified into medium target and large target prediction layers.

[0076] The segmentation mask output by the YOLOV8 segmentation network is a two-dimensional matrix of the same size as the image, which is used to mark different areas or objects in the image. The matrix is ​​usually composed of pixel values, which can be binary (for example, 0 and 1, where 0 represents the background and 1 represents the target object) or multi-valued (for example, different integer values ​​are used to represent different object categories). It is used to clarify which pixels in the image belong to specific objects or regions, thereby separating the target from the background or other irrelevant parts.

[0077] In step S3, the specific process of the PointSeg layer projecting the three-dimensional point cloud to the plane coordinates is as follows:

[0078] S301: Get the intrinsic parameter matrix K of the current camera:

[0079]

[0080] Among them, f x and f y are the focal lengths of the camera in the x and y directions respectively; c x and c y are the coordinates of the center of the image.

[0081] S302: Convert the coordinates (X, Y, Z) of the three-dimensional point in the camera coordinate system to normalized image plane coordinates:

[0082]

[0083] And use the intrinsic parameter matrix K of the current camera to convert the normalized image plane coordinates to pixel coordinates (u, v)

[0084] u=f x ×x'+c x

[0085] v=fy ×y'+c y

[0086] Through the operations of steps S2 and S3, the T-shaped shaft support base can be screened out from the image, such as Figure 4 As shown, the one marked in red is the screened T-shaft bracket base.

[0087] S4: The coordinates (u, v) of the 3D point cloud projected onto the plane are compared and screened with the points in the segmentation mask, and the redundant points are deleted, leaving only the corresponding point cloud in the segmentation mask.

[0088] Through the operation of step S4, the T-shaped shaft support base and the background pattern can be separated in the image, and only the T-shaped shaft support base can be displayed in the image. The separated image is as follows: Figure 5 shown.

[0089] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A point cloud segmentation method for a T-shaped shaft bracket base part based on image guidance, characterized in that: The following steps are involved: S1: Create a T-shaped shaft bracket base part dataset: collect T-shaped shaft bracket base part images and preprocess the images to improve the number of samples in the dataset and the quality of the images; S2: Construct a discriminative feature attention module: input the feature map X of the T-shaped shaft bracket base part image into the three-branch structure and output the feature map; S3: Build an improved YOLOV8 network: Add the discriminative feature attention module constructed in step S2 between the backbone layer and the prediction layer in the YOLOV8 network, and add a PointSeg module after the prediction layer; input the output result of the prediction layer and the point cloud data into the PointSeg module to obtain the coordinates of the three-dimensional point cloud projected onto the plane; S4: The coordinates of the 3D point cloud projected onto the plane are compared and screened with the points in the segmentation mask, and the corresponding point cloud in the segmentation mask is retained.

2. The image-guided point cloud segmentation method for a T-axis bracket base part according to claim 1, characterized in that: In step S1, the step of preprocessing the image includes: flipping the image horizontally and vertically; and adjusting the brightness of the image.

3. The image-guided point cloud segmentation method for a T-axis bracket base part according to claim 2, characterized in that: The adjusting the brightness of the picture includes: increasing the brightness of the picture by 10-20% or decreasing the brightness of the picture by 10-20%.

4. The image-guided point cloud segmentation method for a T-axis bracket base part according to claim 3, characterized in that: The adjusting the brightness of the picture includes: increasing or decreasing the brightness of the picture by 15%.

5. The image-guided point cloud segmentation method for a T-axis bracket base part according to claim 1, characterized in that: Step S2 includes the following steps: S201: Input the feature map X input by the first branch in the three-branch structure into the posture attention module, and output the first branch feature map X1; wherein the feature map X refers to a set of feature vectors generated by convolution after the image of the T-shaped shaft bracket base part is input into the network; S202: performing channel multiplication on the feature map X input by the second branch and the feature map X1 of the first branch, and outputting the feature map X2 of the second branch; S203: Input the second branch feature map X2 into the channel attention module, and output the attention feature map X3; S204: Multiply the feature map X inputted by the third branch by the attention feature map X3 to obtain the third branch feature map X4, which is the output result of the feature attention mechanism.

6. The method for image-guided point cloud segmentation of a T-axis support base part according to claim 5, characterized in that: In step S201, the posture attention module normalizes the feature map X to obtain the horizontal feature X W And the vertical feature X H , as shown in the following formula: X H =Y Avgpool (BatchNorm(Relu(X))) X W =X Avgpool (BatchNorm(Relu(X))) Among them, Y Avgpool is the 1*1 average pooling in the vertical direction, X Avgpool It is the 1*1 average pooling in the horizontal direction, Relu is the activation function, and BatchNorm is the normalization operation; The first branch feature map X1 output includes the horizontal feature X W And the vertical feature X H , as shown in the following formula: X1=X H +X W 。 7. The image-guided point cloud segmentation method for a T-axis support base part according to claim 6, characterized in that: In step S203, in the channel attention module, the position attention mechanism operation is performed on the second branch feature map X2, as shown in the following formula: X3=MC(X2) Among them, MC is the operation of the position attention mechanism.

8. The image-guided point cloud segmentation method for a T-axis bracket base part according to claim 1, characterized in that: The improved YOLOV8 segmentation network also includes: modifying the prediction layers of small, medium and large scales output by the prediction layer into medium target and large target prediction layers.

9. The image-guided point cloud segmentation method for a T-axis support base part according to claim 1, characterized in that: In step S3, the specific process of the PointSeg layer projecting the 3D point cloud to the plane coordinates is as follows: S301: Get the intrinsic parameter matrix K of the current camera: Among them, f x and f y are the focal lengths of the camera in the x and y directions respectively; c x and c y are the coordinates of the center of the image; S302: Convert the coordinates (X, Y, Z) of the three-dimensional point in the camera coordinate system to normalized image plane coordinates: And use the intrinsic parameter matrix K of the current camera to convert the normalized image plane coordinates to pixel coordinates (u, v) u=f x ×x'+c x v=f y ×y'+c y 。 10. The method for image-guided point cloud segmentation of a T-axis support base part according to claim 1, characterized in that: In step S1, the samples in the data set are divided into a training set and a validation set at a ratio of 9:1, which are used to train and validate the model established in the subsequent step S3.

Images