3D point cloud generation, identification and segmentation method

Abstract

The invention provides a 3D point cloud generation, identification and segmentation method, a device comprises a laser, a camera, a motor and a controller, the laser is assembled on the motor, and the controller can control the motor to rotate according to a preset speed so as to drive the laser to rotate and can obtain and store the rotation angle of the motor; the laser is controlled to emit a beam of planar laser; the controller can control the camera to take photos according to different parameters and store photos, and the device belongs to a point cloud generation device of a monocular structured light method, is suitable for point cloud generation in a small range and a short distance, and is high in precision; according to the 3D point cloud generation, identification and segmentation method, the three-dimensional point cloud and the two-dimensional image of the to-be-measured object can be synchronously acquired, the point cloud corresponds to the image, and the identification and segmentation of the three-dimensional point cloud can be realized by identifying and segmenting the two-dimensional image, so that the complexity of 3D point cloud identification and segmentation is remarkably reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional visual imaging, and in particular to a 3D point cloud generation, recognition and segmentation method. Background Art

[0002] Three-dimensional visual imaging refers to the capture or generation of images with three-dimensional spatial information through specific technical means. The core of three-dimensional visual imaging is to obtain and process depth information, so as to reconstruct the three-dimensional structure of an object on a computer or display device. Common methods for achieving three-dimensional visual imaging include: Stereo Vision: Using two cameras to simulate the human eye, shooting objects from different perspectives, and calculating depth information by comparing the pixel differences in the two images. Structured Light: Projecting light of a specific pattern (such as stripes, grids) onto the surface of an object, and calculating the depth and shape of the object by analyzing the deformation of light. Time of Flight (ToF): Emitting light or laser pulses and measuring the reflection time, and calculating the distance based on the speed of light to obtain depth information. In three-dimensional vision, there are many ways to represent three-dimensional information. Common three-dimensional information representation methods include point cloud, triangle mesh, voxel, and depth map. Point cloud is a data structure (set) composed of a large number of three-dimensional coordinate points, and each point represents a position on the surface of an object.

[0003] Point cloud generation method refers to the process of converting data obtained from different sources into point clouds. Point cloud recognition is a technology that extracts useful information from 3D point cloud data and performs recognition and classification. It is a mainstream task in computer 3D vision. For different recognition tasks, the input point cloud needs to be classified, component segmented, or target detected, etc. Point cloud segmentation is to divide the point cloud according to spatial, geometric and texture features so that the point clouds in the same division have similar features.

[0004] Currently, common point cloud generation methods include stereo vision, LiDAR, and monocular structured light. Among them, the depth measurement range and accuracy of the stereo vision method are limited by the distance between the two cameras, and two cameras are required, which is costly. The LiDAR method uses the time of flight (TOF) method to measure depth. TOF is suitable for point cloud generation over a large range and long distance, but has low accuracy in point cloud generation over close distances and small areas.

[0005] At present, point cloud recognition methods are usually based on geometric information or deep learning. The method based on geometric information analyzes and recognizes objects in the point cloud through the geometric features of the point cloud connotation (such as normal vectors, curvature, etc.). This method is usually only applicable to objects with strong regularity or structured scenes, such as buildings. The method based on deep learning is to construct a feature extraction neural network to extract high-dimensional feature representations from the input point cloud data; then, according to different recognition tasks, a recognition network is established to output recognition results based on the extracted feature representations. In some methods, the feature extraction network needs to design corresponding network structures for different recognition tasks, and it is necessary to randomly initialize the learnable network parameters first, and then obtain the final deployable feature extraction network through deep learning training. It can be seen that the training cycle of the feature extraction network in the existing point cloud recognition method based on deep learning is long, and the network scale is large, which reduces the efficiency of point cloud recognition.

[0006] Existing methods for implementing point cloud segmentation include traditional methods based on the geometric features and spatial relationships of point cloud content, such as region growing method, RANSAC algorithm, superbody clustering algorithm, Euclidean algorithm and other algorithms, as well as methods based on deep learning, such as PointNet and PointNet++. First, these methods process point cloud data point by point, which has huge computational overhead and takes a long time. Secondly, the features relied on by the above methods are all from the internal information of the three-dimensional point cloud itself, and no additional information from other sources is effectively used. Therefore, there is a significant room for improvement in the speed and accuracy of point cloud segmentation. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present application proposes a 3D point cloud generation, recognition and segmentation method.

[0008] The present application proposes a 3D point cloud generation, recognition and segmentation method, the device based on which includes a laser, a camera, a motor and a controller, wherein the laser is mounted on the motor, the controller can control the motor to rotate at a preset speed to drive the laser to rotate, and can obtain and store the angle of rotation of the motor to obtain and store the angle of rotation of the laser; the laser is controlled to emit a beam of planar laser; the controller can control the camera to take pictures according to different parameters and store the pictures;

[0009] The method comprises the following steps:

[0010] Step 1, controlling the motor to rotate by the controller to drive the laser to rotate to a zero position, wherein the zero position refers to an initial position, and then turning on the laser to emit a planar laser beam;

[0011] Step 2, keep the camera posture unchanged, control the motor to rotate at a preset speed through the controller, so that the motor drives the laser to rotate, and when the laser rotates, the controller controls the camera to take pictures continuously at a predetermined frequency according to the first shooting parameter to obtain a number of structured light pictures until the laser completely scans the object to be measured; the camera posture remains unchanged during the shooting process; the camera transmits the taken pictures to the controller, and the controller simultaneously records the pictures taken by the camera and the angle of rotation of the laser during shooting; wherein the first shooting parameter ensures that the camera 6 can and can only capture the laser stripes in the image;

[0012] Step 3, assuming that the number of structured light photos taken in step 2 is M, for each structured light photo taken in step 2, the center line of the structured light stripe is extracted after processing, and the coordinate set of all pixel points contained in the center line of the Li-th structured light stripe in the image coordinate system is recorded as Pi, 1≤i≤M; using the camera calibration data, the structured light calibration data and the angle of the structured light rotation, Pi is converted to the world coordinate system to obtain a coordinate set, recorded as Qi; the coordinates in the coordinate sets Q1, Q2, ..., QM in the world coordinate system obtained based on all structured light photos are combined to form a set of point clouds;

[0013] Step 4: Keep the camera posture unchanged, turn off the laser light, and control the camera to take pictures according to the second shooting parameters to obtain a full-view picture of the object to be measured; wherein the second shooting parameters ensure that the camera can take a full-view picture of the object to be measured;

[0014] Step 5: Input the full-view photo of the object to be tested obtained in step 4 into a pre-trained image segmentation neural network. The neural network outputs n sets, where n is a positive integer and is sequentially recorded as A1…Aj…An. Each set Aj contains several pixels in the full-view photo, and these pixels belong to the same semantic Tj.

[0015] Step 6: For each pixel point in each set Aj, find the corresponding point in the point cloud, and form the points in the point cloud into a set Bj, which also belongs to the semantics Tj; finally, the point cloud generated in step 3 is formed into n sets and their corresponding semantics, thus completing the recognition of the point cloud, and then dividing the point cloud into several sub-point clouds according to the semantics to complete the segmentation of the point cloud.

[0016] In some embodiments, in step 3, the processing process includes filtering, denoising, morphological operation, and edge detection processing on the structured light photo in sequence.

[0017] In some embodiments, the laser is an inline laser.

[0018] The beneficial effect of the scheme of the present application lies in that in the above-mentioned 3D point cloud generation, recognition and segmentation method, the device based on it belongs to a point cloud generation device of the monocular structured light method, which is suitable for point cloud generation in a smaller range and at a close distance with higher accuracy; compared with other point cloud generation devices using the monocular structured light method, the present application uses laser rotation scanning to replace mirror group rotation scanning and device translation scanning. Compared with mirror group rotation scanning, the present application reduces the complexity of structured light positioning and improves the point cloud generation accuracy. Compared with device translation scanning, the present application simplifies the motion mechanism and improves the scanning speed; the present application links the recognition and segmentation process of three-dimensional point clouds with the two-dimensional image recognition and segmentation process, and significantly improves the speed and accuracy of 3D point cloud recognition and segmentation with the help of existing two-dimensional image recognition and segmentation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the structure of the device on which the 3D point cloud generation, recognition and segmentation method in the embodiment is based is shown.

[0020] Figure 2 (a) shows the structured light photo taken in step 2 of the embodiment, and (b) is a schematic diagram of the result after processing the structured light photo in step 3.

[0021] Figure 3 The point cloud data diagram of the object to be measured obtained in step 3 of the embodiment is shown.

[0022] Figure 4 The full-view photo of the object to be tested taken in step 4 of the embodiment is shown.

[0023] Figure 5 The figure shows the semantic recognition result of the image segmentation neural network pre-trained in step 5 of the embodiment on the full-view photo of the object to be tested.

[0024] Figure 6 The figure shows the semantic recognition result of the point cloud of the object to be tested in step 6 of the embodiment.

[0025] Figure 7 The figure shows the segmentation result of the point cloud of the object to be measured in step 6 of the embodiment.

[0026] Reference numerals: 1-motor, 2-laser, 3-single chip microcomputer, 4-planar laser, 5-computer, 6-camera, 7-object to be measured. DETAILED DESCRIPTION

[0027] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0028] like Figure 1As shown, the 3D point cloud generation, recognition and segmentation method involved in the present application is based on a device including a laser 2, a camera 6, a motor 1 and a controller. The laser 2 is mounted on the motor 1. The motor 1, the laser 2 and the camera 6 are all connected to the controller. The controller may include a single-chip unit 3 and a computer 5 connected in communication. Specifically, the single-chip unit 3 is respectively connected to the motor 1, the laser 2 and the camera 6, and the single-chip unit 3 sends control signals to the motor 1, the laser 2 and the camera 6 respectively, and the single-chip unit 3 can transmit information to the computer 5. The controller can control the motor 1 to rotate at a preset speed to drive the laser 2 to rotate, and can obtain and store the angle of rotation of the motor 1 to obtain and store the angle of rotation of the laser 2; the laser 2 is controlled to emit a beam of planar laser 4; the controller can control the camera 6 to take pictures according to different parameters and store pictures.

[0029] The 3D point cloud generation, recognition and segmentation method involved in this application includes the following steps:

[0030] Step 1: Control the motor 1 to rotate through the controller to drive the laser 2 to rotate to the zero position, where the zero position refers to the initial position. Any position can be designated as the initial position. Then turn on the laser 2, and the laser 2 emits a beam of planar laser 4.

[0031] Step 2, keep the position of camera 6 unchanged, control the motor 1 to rotate at a preset speed through the controller, so that the motor 1 drives the laser 2 to rotate, and when the laser 2 rotates, the controller controls the camera 6 to take pictures continuously at a predetermined frequency according to the first shooting parameter to obtain a number of structured light pictures until the laser completely scans the object 7 to be measured; the position of camera 6 remains unchanged during the shooting process; the camera 6 transmits the taken pictures to the controller, and the controller simultaneously records the pictures taken by the camera 6 and the angle of rotation of the laser 2 during the shooting; wherein, the first shooting parameter ensures that the camera 6 can and can only capture the laser stripes in the image.

[0032] Step 3, assuming that the number of structured light photos taken in step 2 is M, for each structured light photo taken in step 2, the center line of the structured light stripe is extracted after processing, wherein the processing process includes filtering, denoising, morphological operation, and edge detection processing of the structured light photos in sequence; the coordinate set of all pixel points contained in the center line of the Li-th structured light stripe in the image coordinate system is recorded as Pi, 1≤i≤M; using the camera calibration data, the structured light calibration data and the angle of rotation of the structured light (that is, the angle of rotation of the laser 2), Pi is converted to the world coordinate system to obtain a coordinate set, recorded as Qi; the coordinates in the coordinate sets Q1, Q2, ..., QM in the world coordinate system obtained based on all structured light photos are combined to form a set of point clouds.

[0033] Step 4, keeping the position of the camera 6 unchanged, turning off the light of the laser 2, and controlling the camera 6 to take pictures according to the second shooting parameters to obtain a full-view picture of the object 7 to be measured; wherein the second shooting parameters ensure that the camera 6 can take a full-view picture of the object 7 to be measured.

[0034] Since the camera 6 pose remains unchanged, the points in Pi obtained in step 3 correspond to the pixels in the full photo. Furthermore, each point in the point cloud obtained in step 3 corresponds to a pixel in the full photo.

[0035] Step 5: Input the full-view photo of the object 7 obtained in step 4 into a pre-trained image segmentation neural network. The neural network outputs n sets, where n is a positive integer and is sequentially recorded as A1…Aj…An. Each set Aj (1≤j≤n) contains several pixels in the full-view photo, and these pixels belong to the same semantics Tj.

[0036] Step 6: For each pixel point in each set Aj, find the corresponding point in the point cloud, and form the points in the point cloud into a set Bj, which also belongs to the semantics Tj; finally, the point cloud generated in step 3 is formed into n sets and their corresponding semantics, thus completing the recognition of the point cloud, and then dividing the point cloud into several sub-point clouds according to the semantics to complete the segmentation of the point cloud.

[0037] The 3D point cloud generation, recognition and segmentation method involved in the present application can synchronously collect the three-dimensional point cloud and two-dimensional image of the object to be measured 7, and realize the correspondence between the point cloud and the image, and then can realize the recognition and segmentation of the three-dimensional point cloud by recognizing and segmenting the two-dimensional image, thereby significantly reducing the complexity of 3D point cloud recognition and segmentation and improving efficiency.

[0038] Example

[0039] In the device on which the 3D point cloud generation, recognition and segmentation method is based, the laser 2 adopts a straight-line laser.

[0040] Before step 1 of the 3D point cloud generation, recognition and segmentation method, the following preparations can be done in advance: establish a world coordinate system with a workbench (the object to be measured 7 is placed on the workbench) as a reference point, calibrate the camera and structured light, obtain the external parameter matrix and structured light parameter equation of the camera 6. Set the shooting parameters of the camera 6, including exposure time, exposure gain, aperture size, etc.

[0041] The 3D point cloud generation, recognition and segmentation method comprises the following steps:

[0042] Step 1: Control the motor 1 to rotate through the controller to drive the laser 2 to rotate to the zero position, where the zero position refers to the initial position. Any position can be designated as the initial position. Then turn on the laser 2, and the laser 2 emits a beam of planar laser 4.

[0043] Step 2, keep the position of camera 6 unchanged, control the output shaft of motor 1 to rotate at a preset speed of 2.5° / s through the controller, so that motor 1 drives laser 2 to rotate, and when laser 2 rotates, controller controls camera 6 to take pictures continuously at a predetermined frequency (25 pictures / s) according to the first shooting parameter to obtain a number of structured light pictures until the laser completely scans the object 7 to be measured; the position of camera 6 remains unchanged during the shooting process; camera 6 transmits the taken pictures to the controller, and the controller simultaneously records the pictures taken by camera 6 and the angle of rotation of laser 2 during shooting (compared to the angle rotated at zero position), and the controller stores the position of laser 2 while storing the structured light pictures, so that the structured light pictures correspond to the position of laser 2 one by one; wherein, the first shooting parameter ensures that camera 6 can and can only capture the laser stripes in the image.

[0044] Step 3: Assuming that the number of structured light photos taken in step 2 is M, for each structured light photo taken in step 2, the center line of the structured light stripes is extracted after processing. The processing process includes filtering, denoising, morphological operation, and edge detection processing on the structured light photos in sequence. The original image and the processed image are shown in FIG. Figure 2 As shown; the coordinate set of all pixels contained in the center line of the Li-th structured light stripe in the image coordinate system is recorded as Pi, 1≤i≤M; Pi is converted to the world coordinate system using the camera calibration data, the structured light calibration data and the angle of the structured light (that is, the angle of the laser 2), and a coordinate set is obtained, recorded as Qi; the coordinates in the coordinate sets Q1, Q2, ..., QM in the world coordinate system obtained based on all structured light photos are combined to form a set of point clouds, as shown Figure 3 shown.

[0045] Step 4: Keep the position of the camera 6 unchanged, turn off the light of the laser 2, and control the camera 6 to take pictures according to the second shooting parameters to obtain a full-view picture of the object 7 to be measured, such as Figure 4 As shown; wherein the second shooting parameter ensures that the camera 6 can capture the full view of the object to be measured 7.

[0046] Since the camera 6 pose remains unchanged, the points in Pi obtained in step 3 correspond to the pixels in the full photo. Furthermore, each point in the point cloud obtained in step 3 corresponds to a pixel in the full photo.

[0047] Step 5: Input the full-view photo of the object 7 obtained in step 4 into the pre-trained Unet image segmentation neural network. The neural network outputs 4 sets, which are sequentially recorded as A1…A2…A4. Each set Aj (1≤j≤4) contains several pixels in the full-view photo, and these pixels belong to the same semantics Tj. Figure 5 As shown, the four collections are already in Figure 5 Marked in.

[0048] Step 6: For each pixel in each set Aj, find the corresponding point in the point cloud, and form these points in the point cloud into a set Bj, which also belongs to the semantics Tj; finally, the point cloud generated in step 3 is formed into n sets and their corresponding semantics, such as Figure 6 As shown in the figure, semantics T1, T2, and T3 have been marked in the figure, where semantics T2 represents a workpiece, semantics T3 represents another workpiece, semantics T1 represents a workbench, and semantics T4 represents the mounting hole on the workbench, which is not shown in the figure. At this point, the recognition of the point cloud is completed, and then the point cloud is divided into several sub-point clouds according to the semantics, such as Figure 7 As shown, the point cloud is divided into two sub-point clouds according to semantics T2 and semantics T3, completing the segmentation of the point cloud.

[0049] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solution and concept of the present application within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A 3D point cloud generation, recognition and segmentation method, characterized by: The device includes a laser, a camera, a motor and a controller. The laser is mounted on the motor. The controller can control the motor to rotate at a preset speed to drive the laser to rotate, and can obtain and store the angle of rotation of the motor to obtain and store the angle of rotation of the laser. The laser is controlled to emit a beam of planar laser. The controller can control the camera to take pictures according to different parameters and store the pictures. The method comprises the following steps: Step 1, the controller controls the motor to rotate, so as to drive the laser to rotate to a zero position, wherein the zero position refers to an initial position, and then the laser is turned on, and the laser emits a beam of planar laser; Step 2, keep the camera posture unchanged, control the motor to rotate at a preset speed through the controller, so that the motor drives the laser to rotate, and when the laser rotates, the controller controls the camera to take pictures continuously at a predetermined frequency according to the first shooting parameter to obtain a number of structured light pictures until the laser completely scans the object to be measured; the camera posture remains unchanged during the shooting process; the camera transmits the taken pictures to the controller, and the controller simultaneously records the pictures taken by the camera and the angle of rotation of the laser during shooting; wherein the first shooting parameter ensures that the camera 6 can and can only capture the laser stripes in the image; Step 3, assuming that the number of structured light photos taken in step 2 is M, for each structured light photo taken in step 2, the center line of the structured light stripe is extracted after processing, and the coordinate set of all pixel points contained in the center line of the Li-th structured light stripe in the image coordinate system is recorded as Pi, 1≤i≤M; using the camera calibration data, the structured light calibration data and the angle of the structured light rotation, Pi is converted to the world coordinate system to obtain a coordinate set, recorded as Qi; the coordinates in the coordinate sets Q1, Q2, ..., QM in the world coordinate system obtained based on all structured light photos are combined to form a set of point clouds; Step 4: Keep the camera posture unchanged, turn off the laser light, and control the camera to take pictures according to the second shooting parameters to obtain a full-view picture of the object to be measured; wherein the second shooting parameters ensure that the camera can take a full-view picture of the object to be measured; Step 5: Input the full-view photo of the object to be tested obtained in step 4 into a pre-trained image segmentation neural network. The neural network outputs n sets, where n is a positive integer and is sequentially recorded as A1…Aj…An. Each set Aj contains several pixels in the full-view photo, and these pixels belong to the same semantic Tj. Step 6: For each pixel point in each set Aj, find the corresponding point in the point cloud, and form the points in the point cloud into a set Bj, which also belongs to the semantics Tj; finally, the point cloud generated in step 3 is formed into n sets and their corresponding semantics, thus completing the recognition of the point cloud, and then dividing the point cloud into several sub-point clouds according to the semantics to complete the segmentation of the point cloud.

2. The 3D point cloud generation, recognition and segmentation method according to claim 1, characterized in that: In step 3, the processing process includes filtering, denoising, morphological operation, and edge detection processing on the structured light photo in sequence.

3. The 3D point cloud generation, recognition and segmentation method according to claim 1, characterized in that: The laser is a straight-line laser.

Images