Three-dimensional point cloud semantic segmentation method and device, equipment and storage medium

Abstract

The invention discloses a three-dimensional point cloud semantic segmentation method based on deep learning, which is used for solving the problems of low segmentation precision and poor segmentation efficiency of the existing three-dimensional point cloud semantic segmentation task. The method comprises the steps of obtaining a depth map corresponding to a to-be-detected object; according to the depth map, generating point cloud data corresponding to the to-be-detected object; inputting the depth map into a pre-trained target detection model, and determining a candidate detection area corresponding to the to-be-detected object through the target detection model; according to the point cloud data corresponding to the to-be-detected object, determining candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the to-be-detected object; inputting the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; and mapping the semantic segmentation result to the point cloud data to realize defect detection of the to-be-detected object.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium and computer program product for three-dimensional point cloud semantic segmentation based on deep learning. Background Art

[0002] With the application of 3D sensors such as LiDAR in unmanned driving and robotics projects, 3D data has received increasing attention from academia and industry. Point cloud is an important form of 3D data representation. It is obtained through direct measurement, similar to the way humans observe the world, and can retain the true 3D structural information of objects to the greatest extent.

[0003] Among them, 3D point cloud semantic segmentation is the basic task in computer vision tasks such as autonomous driving navigation, industrial automatic control and product defect detection. It is also a research hotspot in 3D computer vision and deep learning.

[0004] Specifically, a 3D point cloud refers to a data set of points in a certain coordinate system. In addition to the 3D geometric position, point cloud data may also contain other rich information, such as color, classification value, intensity value, and time. The semantic segmentation of point clouds refers to outputting the category value of each point in the input point cloud based on the 3D spatial coordinates, RGB values, and other data of the input point cloud.

[0005] However, in practical applications, the point cloud actually obtained contains a lot of noise, which will reduce the accuracy of point cloud semantic segmentation. On the other hand, point cloud data is usually huge in volume, and each scan may generate millions or even billions of points, which will result in a lot of computing resources being consumed when training on massive data.

[0006] It can be seen that how to improve the segmentation accuracy and processing efficiency of the semantic segmentation task of three-dimensional point clouds has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The embodiment of the present application provides a three-dimensional point cloud semantic segmentation method based on deep learning, which is used to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing three-dimensional point cloud semantic segmentation tasks.

[0008] The embodiment of the present application also provides a three-dimensional point cloud semantic segmentation device based on deep learning, which is used to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing three-dimensional point cloud semantic segmentation tasks.

[0009] The embodiment of the present application also provides a three-dimensional point cloud semantic segmentation device based on deep learning, which is used to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing three-dimensional point cloud semantic segmentation tasks.

[0010] The embodiment of the present application also provides a computer-readable storage medium to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing three-dimensional point cloud semantic segmentation tasks.

[0011] The embodiment of the present application also provides a computer program product to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing three-dimensional point cloud semantic segmentation tasks.

[0012] The present application embodiment adopts the following technical solutions:

[0013] A three-dimensional point cloud semantic segmentation method based on deep learning, comprising: obtaining a depth map corresponding to an object to be detected; generating point cloud data corresponding to the object to be detected based on the depth map; inputting the depth map into a pre-trained target detection model, and determining a candidate detection area corresponding to the object to be detected through the target detection model; determining candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected based on the point cloud data corresponding to the object to be detected; inputting the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data based on the point cloud semantic segmentation model; mapping the semantic segmentation result to the point cloud data to realize defect detection of the object to be detected.

[0014] A three-dimensional point cloud semantic segmentation device based on deep learning, comprising: a depth map acquisition unit, used to acquire a depth map corresponding to an object to be detected; a depth map conversion unit, used to generate point cloud data corresponding to the object to be detected based on the depth map; a candidate area recognition unit, used to input the depth map into a pre-trained target detection model, and determine the candidate detection area corresponding to the object to be detected through the target detection model; a candidate area conversion unit, used to determine, based on the point cloud data corresponding to the object to be detected, the candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected; a semantic segmentation unit, used to input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain the semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; a prediction unit, used to map the semantic segmentation result to the point cloud data, so as to realize defect detection of the object to be detected.

[0015] A 3D point cloud semantic segmentation device based on deep learning, comprising:

[0016] A processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to perform the following operations: obtain a depth map corresponding to an object to be detected; generate point cloud data corresponding to the object to be detected based on the depth map; input the depth map into a pre-trained target detection model, and determine a candidate detection area corresponding to the object to be detected through the target detection model; determine candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected based on the point cloud data corresponding to the object to be detected; input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain a semantic segmentation result corresponding to the candidate point cloud data based on the point cloud semantic segmentation model; map the semantic segmentation result to the point cloud data to realize defect detection of the object to be detected.

[0017] A computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple application programs, enable the electronic device to perform the following operations: obtain a depth map corresponding to an object to be detected; generate point cloud data corresponding to the object to be detected based on the depth map; input the depth map into a pre-trained target detection model, and determine a candidate detection area corresponding to the object to be detected through the target detection model; determine candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected based on the point cloud data corresponding to the object to be detected; input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain semantic segmentation results corresponding to the candidate point cloud data based on the point cloud semantic segmentation model; map the semantic segmentation results to the point cloud data to achieve defect detection of the object to be detected.

[0018] A computer program product comprises a computer program, which, when executed by a processor, implements the following: obtaining a depth map corresponding to an object to be detected; generating point cloud data corresponding to the object to be detected based on the depth map; inputting the depth map into a pre-trained target detection model, and determining a candidate detection area corresponding to the object to be detected through the target detection model; determining, based on the point cloud data corresponding to the object to be detected, candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected; inputting the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data based on the point cloud semantic segmentation model; and mapping the semantic segmentation result to the point cloud data to implement defect detection of the object to be detected.

[0019] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0020] By using the deep learning-based three-dimensional point cloud semantic segmentation method provided in the embodiment of the present application, when performing defect detection on an object by executing the three-dimensional point cloud semantic segmentation task, first, a depth map corresponding to the object to be detected can be obtained, and based on the obtained depth map, point cloud data corresponding to the object to be detected can be generated; then, the depth map is input into a pre-trained target detection model, and the candidate detection area corresponding to the object to be detected is determined through the target detection model, so as to realize the first stage of preliminary positioning of the defects of the object to be detected on the grayscale image; next, based on the point cloud data corresponding to the object to be detected, the candidate point cloud data corresponding to the candidate detection area preliminarily located in the first stage is determined, and the candidate point cloud data is input into the pre-trained point cloud semantic segmentation model, and the semantic segmentation results corresponding to the candidate point cloud data are obtained according to the point cloud semantic segmentation model, and finally, the semantic segmentation results are mapped to the point cloud data to realize defect detection of the object to be detected. By adopting the deep learning-based 3D point cloud semantic segmentation method provided in the embodiment of the present application, on the one hand, through the two stages of target detection and point cloud semantic segmentation, it is possible to accurately identify and segment small defects in the object to be detected, greatly improving the accuracy of the 3D point cloud semantic segmentation task; on the other hand, through the target detection operation performed on the two-dimensional level, the calculation and analysis scope of the 3D point cloud semantic segmentation task can be quickly reduced, effectively reducing the time consumption in the model calculation process and improving the efficiency of the model in the training and reasoning stages; finally, due to the preliminary screening of the target detection link in the first stage, the amount of data to be processed in the 3D point cloud semantic segmentation task is reduced, and this method can be implemented on lower-specification hardware, reducing the company's investment cost in equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic diagram of a specific process of a 3D point cloud semantic segmentation method based on deep learning provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the specific structure of a three-dimensional point cloud semantic segmentation device provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the specific structure of a three-dimensional point cloud semantic segmentation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0027] The embodiment of the present application provides a deep learning-based 3D point cloud semantic segmentation method to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing 3D point cloud semantic segmentation tasks.

[0028] The executor of the deep learning-based three-dimensional point cloud semantic segmentation method provided in the embodiment of the present application may be, but is not limited to, at least one of a visual recognition server, an autonomous driving navigation server, an automated control server, or a product defect detection server; in addition, the executor of the method may also be an application (Application, APP) running on these servers or the system itself.

[0029] For ease of description, the following takes the method being performed by a product defect detection system as an example to introduce the implementation of the method. It can be understood that the method being performed by a product defect detection system is only an exemplary description and should not be understood as a limitation on the method.

[0030] The specific implementation flow chart of the deep learning-based 3D point cloud semantic segmentation method provided in this application is as follows: Figure 1 As shown, it mainly includes the following steps:

[0031] Step 11, obtaining a depth map corresponding to the object to be detected;

[0032] Specifically, in an embodiment of the present application, a specific camera model can be used to capture images of the object to be detected to obtain image information of the object to be detected at multiple angles, and then one of the angles is designated as the main viewing angle, and the other angles are used as side viewing angles. The original depth map under the side viewing angle is converted to the depth reference system of the main viewing angle, and the original depth map under the main viewing angle is completed to obtain the depth map corresponding to the object to be detected.

[0033] It should be noted that the camera model used to collect the depth map can be a perspective camera model, a cone camera model, or an orthographic camera model, etc. The embodiment of the present application does not limit the specific camera model used when collecting the depth map. In addition, it should be noted that the number of images collected for the object to be detected can be arbitrary while ensuring that the shooting angles do not overlap.

[0034] In addition, when collecting depth maps, the camera models can be arranged in the form of a camera array, for example, they can be arranged in a parallel camera array, in which the main optical axes of each camera are perpendicular to the plane formed by the scene; or they can be arranged in a convergent camera array, in which the main optical axes of each camera converge at a common point in the scene; or they can be arranged in a staggered camera array, in which the main optical axes of each camera converge at a common point in the scene and the display plane displays the same range through calculation. The embodiments of the present application do not specifically limit the camera array in which the camera models are arranged when collecting depth maps.

[0035] Step 12, generating point cloud data corresponding to the object to be detected according to the depth map obtained by executing step 11;

[0036] In the embodiment of the present application, the product defect detection system can convert the depth map obtained by executing step 11 into point cloud data corresponding to the object to be detected through the following sub-steps, specifically including:

[0037] Sub-step 1201, converting the depth image corresponding to the object to be detected into a grayscale image;

[0038] In the embodiment of the present application, by converting the depth map into a grayscale map, the background of the image corresponding to the object to be detected can be weakened and the contour information of the object to be detected in the image can be enhanced.

[0039] In one implementation, a global threshold method or a local threshold method may be used to convert the depth map into a grayscale map.

[0040] In the embodiment of the present application, a local threshold method is used to convert a depth map into a grayscale image. First, data preprocessing can be performed on the depth map to segment the depth map into blocks. Secondly, any depth sub-image obtained after segmentation is selected, and the sum of the weighted results of the depth values ​​of each pixel in the depth sub-image area is used as the conversion threshold. According to the conversion threshold, the depth values ​​of the pixels in the depth sub-image area are converted into grayscale values. The above steps are repeated until all the segmented areas of the depth map are converted into grayscale images, and the processing operation of converting the depth map into the grayscale image is completed.

[0041] Sub-step 1202, obtaining depth map data corresponding to the depth map;

[0042] In the embodiment of the present application, the depth map data may refer to the coordinate information of the depth map on the X, Y and Z axes, and the coordinate information is automatically collected and saved by the camera model when executing step 11 to collect the depth map.

[0043] Sub-step 1203, determining the Z-axis value corresponding to the depth map according to the depth map data;

[0044] Sub-step 1204: generating point cloud data corresponding to the object to be detected according to the grayscale image and the Z-axis value.

[0045] The Z-axis value in the collected depth map data is stored in a grayscale image with the same width and height as the point cloud, realizing the conversion of 2D grayscale image to 3D point cloud data.

[0046] In the embodiment of the present application, the conversion from 2D grayscale image to 3D point cloud data can be realized by the following conversion formulas [1] to [3]:

[0047] z=(zc-offsetZ)*resZ [1]

[0048] x=(u-Cx)*z*inv_fx [2]

[0049] y=(v-Cy)*z*inv_fy [3]

[0050] Among them, resZ is the data stored when the depth map is collected, indicating the resolution in the Z-axis direction; offsetZ is the offset value in the Z-axis direction; zc is the original depth value in the camera coordinate system; inv_fx and inv_fy represent the inverse of the focal length of the camera on the X-axis and Y-axis; Cx and Cy represent the aperture center X and Y of the camera respectively; u and v represent the corresponding coordinates of x and y on the image plane respectively.

[0051] Step 13, inputting the depth map collected by executing step 11 into a pre-trained target detection model, and determining a candidate detection area corresponding to the object to be detected through the target detection model;

[0052] In the embodiment of the present application, the target detection model can be trained by the following method, which specifically includes the following sub-steps:

[0053] Sub-step 1301, obtaining a second depth map corresponding to the detected object;

[0054] In an embodiment of the present application, depth maps corresponding to objects whose defects have been detected and identified may be used as training samples, and second depth maps corresponding to the detected objects may be obtained.

[0055] Sub-step 1302, marking defects on the second depth map according to the defect detection result of the detected object to obtain a marked depth map;

[0056] Among them, the defect detection results are used to represent the location information of defects existing in the detected object. In an embodiment of the present application, the product defect detection system can mark the second depth map corresponding to the detected object based on these defect detection results, mark the defect locations therein, that is, the target detection object, and thus obtain the marked depth map.

[0057] Sub-step 1303, constructing a first training sample set according to the annotated depth map;

[0058] Through the above sub-steps 1301 to 1302, multiple annotated depth maps are determined respectively, and these annotated depth maps are converted into grayscale images. The annotated grayscale images are randomly cropped, rotated, and folded to expand and enhance the data, and a first training sample set for training the target detection model is constructed based on the expanded and enhanced data.

[0059] Sub-step 1304: training an initial target detection model according to the first training sample set, and adjusting model parameters of the initial target detection model according to a first loss function to obtain a target detection model.

[0060] Input each first training sample in the first training sample set into the initial target detection model respectively to obtain a first prediction result output by the initial target detection model, wherein the first prediction result at least includes: a predicted target detection box for the defect, and then construct a first loss function based on the difference between the predicted target detection box and the labeled target box, and adjust the model parameters of the initial target detection model according to the first loss function, thereby completing the training of the target detection model.

[0061] The embodiment of the present application does not limit the specific training method of the target detection model. It only needs to ensure that the trained target detection model can extract the features of the object to be detected and perform information fusion, and then predict the candidate identification region (Region of Interest, RoI) of the object to be detected.

[0062] The product defect detection system can first input the depth map obtained by executing step 11 into a pre-trained target detection model, and then use the target detection model to roughly locate the defects of the object to be detected, and determine the candidate detection area RoI corresponding to the object to be detected based on the rough positioning result of the defect.

[0063] Step 14, determining candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected according to the point cloud data corresponding to the object to be detected;

[0064] In an embodiment of the present application, the product defect detection system can resize the candidate identification area RoI image output by the target detection model, and map the pixels in the adjusted candidate identification area RoI image back to the point cloud data generated by executing step 12, thereby determining the candidate point cloud data corresponding to the candidate detection area RoI, and achieving accurate positioning of the candidate identification area RoI in three-dimensional space.

[0065] Step 15, inputting the candidate point cloud data obtained by executing step 14 into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model;

[0066] In an embodiment of the present application, the point cloud semantic segmentation model can be trained by the following method, which specifically includes the following sub-steps:

[0067] Sub-step 1501, obtaining second point cloud data corresponding to the detected object;

[0068] In an embodiment of the present application, point cloud data corresponding to an object whose defects have been detected and identified may be obtained as a training sample.

[0069] Sub-step 1502, performing defect labeling on the second point cloud data according to the defect detection result of the detected object to obtain labeled point cloud data;

[0070] Among them, the defect detection results are used to represent the location information of defects existing in the detected objects. In the embodiment of the present application, the product defect detection system can annotate the point cloud data corresponding to the detected objects according to these defect detection results, mark the defect locations therein, that is, the target detection objects, and then obtain the annotated point cloud data.

[0071] Sub-step 1503, constructing a second training sample set according to the labeled point cloud data;

[0072] Through the above sub-steps 1501 to 1502, multiple labeled point cloud data are determined respectively, and these labeled point cloud data are randomly cropped, rotated, folded, etc. to complete data expansion and enhancement, and a second training sample set for training the point cloud semantic segmentation model is constructed based on the expanded and enhanced data.

[0073] Sub-step 1504: training the initial point cloud semantic segmentation model according to the second training sample set, and adjusting the model parameters of the initial point cloud semantic segmentation model according to the second loss function to obtain the point cloud semantic segmentation model.

[0074] Input each second training sample in the second training sample set into the initial point cloud semantic segmentation model respectively to obtain the prediction result output by the initial point cloud semantic segmentation model, wherein the prediction result at least includes: a predicted target detection box for the defect, and then construct a second loss function according to the difference between the predicted target detection box and the labeled target box, and adjust the model parameters of the initial point cloud semantic segmentation model according to the second loss function, thereby completing the training of the point cloud semantic segmentation model.

[0075] The embodiment of the present application does not limit the specific training method of the point cloud semantic segmentation model.

[0076] The product defect detection system can first input the point cloud data obtained by executing step 12 into a pre-trained point cloud semantic segmentation model, and then use the point cloud semantic segmentation model to perform semantic segmentation processing on the candidate recognition area RoI data after the defect of the object to be detected is roughly located, so as to obtain the defect mask and category information within the candidate recognition area RoI.

[0077] Step 16, mapping the semantic segmentation result to the point cloud data to achieve defect detection of the object to be detected.

[0078] Specifically, the product defect detection system can identify defect points in the point cloud data based on the defect location data obtained by executing step 15, and mark these defect points; at the same time, identify all background points in the point cloud data, that is, those points that are not marked as defects, and set them to a predefined zero value to distinguish between background points and defect points; finally, merge the marked background points and defect points to complete the mapping of the prediction results, thereby realizing defect detection of the object to be detected.

[0079] By using the deep learning-based three-dimensional point cloud semantic segmentation method provided in the embodiment of the present application, when performing defect detection on an object by executing the three-dimensional point cloud semantic segmentation task, first, a depth map corresponding to the object to be detected can be obtained, and based on the obtained depth map, point cloud data corresponding to the object to be detected can be generated; then, the depth map is input into a pre-trained target detection model, and the candidate detection area corresponding to the object to be detected is determined through the target detection model, so as to realize the first stage of preliminary positioning of the defects of the object to be detected on the grayscale image; next, based on the point cloud data corresponding to the object to be detected, the candidate point cloud data corresponding to the candidate detection area preliminarily located in the first stage is determined, and the candidate point cloud data is input into the pre-trained point cloud semantic segmentation model, and the semantic segmentation results corresponding to the candidate point cloud data are obtained according to the point cloud semantic segmentation model, and finally, the semantic segmentation results are mapped to the point cloud data to realize defect detection of the object to be detected. By adopting the deep learning-based 3D point cloud semantic segmentation method provided in the embodiment of the present application, on the one hand, through the two stages of target detection and point cloud semantic segmentation, it is possible to accurately identify and segment small defects in the object to be detected, greatly improving the accuracy of the 3D point cloud semantic segmentation task; on the other hand, through the target detection operation performed on the two-dimensional level, the calculation and analysis scope of the 3D point cloud semantic segmentation task can be quickly reduced, effectively reducing the time consumption in the model calculation process and improving the efficiency of the model in the training and reasoning stages; finally, due to the preliminary screening of the target detection link in the first stage, the amount of data to be processed in the 3D point cloud semantic segmentation task is reduced, and this method can be implemented on lower-specification hardware, reducing the company's investment cost in equipment.

[0080] In one embodiment, the present application also provides a 3D point cloud semantic segmentation device based on deep learning, which is used to solve the problems of low segmentation accuracy and poor segmentation efficiency in existing 3D point cloud semantic segmentation tasks. Figure 2 As shown, it includes: a depth map acquisition unit 21, a depth map conversion unit 22, a candidate region identification unit 23, a candidate region conversion unit 24, a semantic segmentation unit 25 and a prediction unit 26.

[0081] Wherein, the depth map acquisition unit 21 is used to acquire the depth map corresponding to the object to be detected;

[0082] A depth map conversion unit 22, configured to generate point cloud data corresponding to the object to be detected according to the depth map;

[0083] A candidate region identification unit 23 is used to input the depth map into a pre-trained target detection model, and determine a candidate detection region corresponding to the object to be detected through the target detection model;

[0084] A candidate region conversion unit 24 is used to determine candidate point cloud data corresponding to the candidate detection region in the point cloud data corresponding to the object to be detected according to the point cloud data corresponding to the object to be detected;

[0085] The semantic segmentation unit 25 is used to input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model;

[0086] The prediction unit 26 is used to map the semantic segmentation result to the point cloud data to achieve defect detection of the object to be detected.

[0087] In one embodiment, the depth map conversion unit 22 is specifically used to: convert the depth map corresponding to the object to be detected into a grayscale image; obtain depth map data corresponding to the depth map; determine the Z-axis value corresponding to the depth map based on the depth map data; and generate point cloud data corresponding to the object to be detected based on the grayscale image and the Z-axis value.

[0088] In one embodiment, it also includes a target detection model training unit, which is specifically used to: obtain a second depth map corresponding to a detected object; annotate the second depth map with defects based on the defect detection result of the detected object to obtain a labeled depth map; construct a first training sample set based on the labeled depth map; train an initial target detection model based on the first training sample set, and adjust the model parameters of the initial target detection model according to a first loss function to obtain a target detection model.

[0089] In one embodiment, it also includes a point cloud semantic segmentation model training unit, which is specifically used to: obtain second point cloud data corresponding to the detected object; annotate the second point cloud data with defects according to the defect detection result of the detected object to obtain annotated point cloud data; construct a second training sample set according to the annotated point cloud data; train the initial point cloud semantic segmentation model according to the second training sample set, and adjust the model parameters of the initial point cloud semantic segmentation model according to a second loss function to obtain a point cloud semantic segmentation model.

[0090] In one implementation, the candidate region conversion unit 24 is specifically configured to: obtain region coordinates corresponding to the candidate detection region; and determine candidate point cloud data corresponding to the candidate detection region according to the region coordinates and the point cloud data.

[0091] In one embodiment, the prediction unit 26 is specifically used to: determine the background points in the point cloud data; assign values ​​to the background points in the point cloud data according to the semantic segmentation result to obtain second point cloud data; map the semantic segmentation result to the second point cloud data to achieve defect detection of the object to be detected.

[0092] By using the deep learning-based three-dimensional point cloud semantic segmentation device provided in the embodiment of the present application, when performing defect detection on an object by executing the three-dimensional point cloud semantic segmentation task, the depth map corresponding to the object to be detected can be first obtained, and based on the obtained depth map, point cloud data corresponding to the object to be detected can be generated; then the depth map is input into a pre-trained target detection model, and the candidate detection area corresponding to the object to be detected is determined through the target detection model, so as to realize the first stage of preliminary positioning of the defects of the object to be detected on the grayscale image; next, based on the point cloud data corresponding to the object to be detected, the candidate point cloud data corresponding to the candidate detection area preliminarily located in the first stage is determined, and the candidate point cloud data is input into the pre-trained point cloud semantic segmentation model, and the semantic segmentation results corresponding to the candidate point cloud data are obtained according to the point cloud semantic segmentation model, and finally the semantic segmentation results are mapped to the point cloud data to realize defect detection of the object to be detected. By adopting the deep learning-based 3D point cloud semantic segmentation method provided in the embodiment of the present application, on the one hand, through the two stages of target detection and point cloud semantic segmentation, it is possible to accurately identify and segment small defects in the object to be detected, greatly improving the accuracy of the 3D point cloud semantic segmentation task; on the other hand, through the target detection operation performed on the two-dimensional level, the calculation and analysis scope of the 3D point cloud semantic segmentation task can be quickly reduced, effectively reducing the time consumption in the model calculation process and improving the efficiency of the model in the training and reasoning stages; finally, due to the preliminary screening of the target detection link in the first stage, the amount of data to be processed in the 3D point cloud semantic segmentation task is reduced, and this method can be implemented on lower-specification hardware, reducing the company's investment cost in equipment.

[0093] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0094] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0095] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0096] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a three-dimensional point cloud semantic segmentation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: obtain the depth map corresponding to the object to be detected; generate the point cloud data corresponding to the object to be detected according to the depth map; input the depth map into a pre-trained target detection model, and determine the candidate detection area corresponding to the object to be detected through the target detection model; determine the candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected according to the point cloud data corresponding to the object to be detected; input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain the semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; map the semantic segmentation result to the point cloud data to realize defect detection of the object to be detected.

[0097] The above application Figure 3The method performed by the electronic device for semantic segmentation of three-dimensional point clouds disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0098] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0099] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0100] Acquire a depth map corresponding to the object to be detected; generate point cloud data corresponding to the object to be detected based on the depth map; input the depth map into a pre-trained target detection model, and determine the candidate detection area corresponding to the object to be detected through the target detection model; determine the candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected based on the point cloud data corresponding to the object to be detected; input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain the semantic segmentation results corresponding to the candidate point cloud data based on the point cloud semantic segmentation model; map the semantic segmentation results to the point cloud data to realize defect detection of the object to be detected.

[0101] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0109] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0110] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A 3D point cloud semantic segmentation method based on deep learning, characterized in that: include: Obtain a depth map corresponding to the object to be detected; Generating point cloud data corresponding to the object to be detected according to the depth map; Inputting the depth map into a pre-trained target detection model, and determining a candidate detection area corresponding to the object to be detected through the target detection model; Determine, according to the point cloud data corresponding to the object to be detected, candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected; Inputting the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; The semantic segmentation result is mapped to the point cloud data to achieve defect detection of the object to be detected.

2. The method according to claim 1, characterized in that Generating point cloud data corresponding to the object to be detected according to the depth map specifically includes: Converting the depth map corresponding to the object to be detected into a grayscale map; Obtaining depth map data corresponding to the depth map; Determine a Z-axis value corresponding to the depth map according to the depth map data; According to the grayscale image and the Z-axis value, point cloud data corresponding to the object to be detected is generated.

3. The method according to claim 1, characterized in that Pre-training the target detection model specifically includes: Obtaining a second depth map corresponding to the detected object; According to the defect detection result of the detected object, the second depth map is marked with defects to obtain a marked depth map; Constructing a first training sample set according to the annotated depth map; An initial target detection model is trained according to the first training sample set, and model parameters of the initial target detection model are adjusted according to a first loss function to obtain a target detection model.

4. The method according to claim 1, characterized in that Pre-training the point cloud semantic segmentation model specifically includes: Acquire second point cloud data corresponding to the detected object; According to the defect detection result of the detected object, defect marking is performed on the second point cloud data to obtain marked point cloud data; Constructing a second training sample set according to the labeled point cloud data; The initial point cloud semantic segmentation model is trained according to the second training sample set, and the model parameters of the initial point cloud semantic segmentation model are adjusted according to the second loss function to obtain the point cloud semantic segmentation model.

5. The method according to claim 2, characterized in that: The step of determining, according to the point cloud data corresponding to the object to be detected, candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected specifically includes: Obtaining the region coordinates corresponding to the candidate detection region; Determine candidate point cloud data corresponding to the candidate detection area according to the area coordinates and the point cloud data.

6. The method according to claim 1, characterized in that Mapping the semantic segmentation result to the point cloud data to implement defect detection of the object to be detected specifically includes: Determining background points in the point cloud data; Assigning values ​​to background points in the point cloud data according to the semantic segmentation result to obtain second point cloud data; The semantic segmentation result is mapped to the second point cloud data to achieve defect detection of the object to be detected.

7. A 3D point cloud semantic segmentation device based on deep learning, characterized in that: include: A depth map acquisition unit, used to acquire a depth map corresponding to the object to be detected; A depth map conversion unit, used to generate point cloud data corresponding to the object to be detected according to the depth map; A candidate region identification unit, configured to input the depth map into a pre-trained target detection model, and determine a candidate detection region corresponding to the object to be detected through the target detection model; A candidate region conversion unit, configured to determine, based on the point cloud data corresponding to the object to be detected, candidate point cloud data corresponding to the candidate detection region in the point cloud data corresponding to the object to be detected; A semantic segmentation unit, used to input the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtain a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; A prediction unit is used to map the semantic segmentation result to the point cloud data to achieve defect detection of the object to be detected.

8. A 3D point cloud semantic segmentation device based on deep learning, comprising: processor; a memory arranged to store computer executable instructions which, when executed, cause the processor to: Obtain a depth map corresponding to the object to be detected; Generating point cloud data corresponding to the object to be detected according to the depth map; Inputting the depth map into a pre-trained target detection model, and determining a candidate detection area corresponding to the object to be detected through the target detection model; Determine, according to the point cloud data corresponding to the object to be detected, candidate point cloud data corresponding to the candidate detection area in the point cloud data corresponding to the object to be detected; Inputting the candidate point cloud data into a pre-trained point cloud semantic segmentation model, and obtaining a semantic segmentation result corresponding to the candidate point cloud data according to the point cloud semantic segmentation model; The semantic segmentation result is mapped to the point cloud data to achieve defect detection of the object to be detected.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, enables the electronic device to perform the deep learning-based three-dimensional point cloud semantic segmentation method as described in any one of claims 1-6.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the deep learning-based three-dimensional point cloud semantic segmentation method as described in any one of claims 1 to 6.

Images