Point cloud edge detection method and device

Abstract

The present disclosure proposes a point cloud edge detection method and device, which relates to the field of computer science. By using point cloud training data, such as a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, the generative network and the discriminant network of the generative adversarial network are iteratively trained, the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, and the discriminant network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, until the generative network can produce a false sample that makes it difficult for the discriminant network to identify the authenticity, and then, the generative network in the trained generative adversarial network is used to perform edge detection on the second input point cloud to be detected, thereby realizing a point cloud edge detection scheme based on the generative adversarial network, which can directly perform edge detection on a three-dimensional point cloud, avoid information loss in the process of point cloud dimensionality reduction, and is conducive to improving the accuracy of point cloud edge detection.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a point cloud edge detection method and device. Background Art

[0002] A point cloud is a collection of points in three-dimensional space. The attributes of the points may include, for example, coordinate position, reflection intensity, and color information.

[0003] The purpose of point cloud edge detection is to determine which points in the point cloud fall on the edge. A related technology projects a 3D point cloud onto a 2D plane to form a 2D image, extracts the image edge using an image edge extraction method, and then maps the image pixels to the 3D point cloud points to obtain the point cloud edge. Summary of the invention

[0004] The inventors found that the related technology suffers from information loss when converting point clouds from three dimensions to two dimensions, making it difficult to extract complete edges and resulting in poor accuracy.

[0005] In view of this, the present invention proposes a point cloud edge detection scheme based on a generative adversarial network, which can directly perform edge detection on three-dimensional point clouds, avoid information loss in the process of point cloud dimensionality reduction, and help improve the accuracy of point cloud edge detection.

[0006] Some embodiments of the present disclosure provide a point cloud edge detection method, including:

[0007] The generative adversarial network is trained using point cloud training data, wherein the generative adversarial network includes a generative network and a discriminative network, and the point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, wherein the training process includes: the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, the discriminative network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, and through iterative training of the generative adversarial network, the difference between the true probability and the false probability in the discrimination result of the discriminative network is made smaller than a preset value;

[0008] The generative network in the trained generative adversarial network is used to perform edge detection on the second input point cloud to be detected.

[0009] In some embodiments, the convolution layer of the generative network or the discriminative network downsamples the point cloud input to the convolution layer into point clouds of multiple granularities, and the point clouds of each granularity are independently convolved, and the results of each operation are cascaded into a point cloud.

[0010] In some embodiments, the convolution layer of the generative network or the discriminative network is provided with a plurality of convolution branches of different scales; the cascade layer of the generative network or the discriminative network keeps the point cloud position unchanged and connects a plurality of convolution branches in series.

[0011] In some embodiments, the parameter N of the first convolution layer in different convolution branches is different, and the parameter N of the last convolution layer in different convolution branches is the same, where the parameter N represents the number of points in the output point cloud.

[0012] In some embodiments, the cascade layer keeps the parameter N of the convolution layer of the last layer in each convolution branch unchanged, and superimposes the parameter C of the convolution layer of the last layer in each convolution branch, where the parameter C represents the number of feature channels of the points in the output point cloud.

[0013] In some embodiments, the results of edge detection include:

[0014] The probability of whether each point of the second input point cloud is an edge point;

[0015] or,

[0016] The probability of the semantic category of each point in the second input point cloud.

[0017] In some embodiments, the second input point cloud is obtained by thinning the original point cloud to be detected.

[0018] Some embodiments of the present disclosure provide a point cloud edge detection device, including:

[0019] Memory; and

[0020] A processor coupled to the memory, wherein the processor is configured to execute the point cloud edge detection method of any one of the aforementioned embodiments based on instructions stored in the memory.

[0021] Some embodiments of the present disclosure provide a point cloud edge detection device, including:

[0022] A training unit is configured to train a generative adversarial network using point cloud training data, wherein the generative adversarial network includes a generative network and a discriminative network, and the point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, wherein the training process includes: the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, the discriminative network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, and through iterative training of the generative adversarial network, the difference between the true probability and the false probability in the discrimination result of the discriminative network is made smaller than a preset value;

[0023] The detection unit is configured to use the generative network in the trained generative adversarial network to perform edge detection on the second input point cloud to be detected.

[0024] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the point cloud edge detection method of any of the aforementioned embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0026] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0027] Figure 1 The present invention is a flowchart of a point cloud edge detection method according to some embodiments of the present invention.

[0028] Figure 2 A schematic diagram of the structure of a generation network according to some embodiments of the present disclosure.

[0029] Figure 3 Schematic diagram of the structure of the discriminant network of some embodiments of the present disclosure.

[0030] Figure 4 It is a schematic diagram of the structure of a point cloud edge detection device according to some embodiments of the present disclosure.

[0031] Figure 5 It is a schematic diagram of the structure of a point cloud edge detection device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0033] The present invention implements a point cloud edge detection solution based on a generative adversarial network, which can directly detect the edge of a three-dimensional point cloud, avoid information loss during the point cloud dimensionality reduction process, and help improve the accuracy of point cloud edge detection. In addition, compared with convolutional neural networks, generative adversarial networks can better complete tasks with small samples.

[0034] The generative adversarial network includes a generative network (set as G) and a discriminative network (set as D). The generative network G generates a false sample G(z) (such as the edge of the point cloud) based on the observed information z (such as the laser point cloud), and the discriminative network D predicts the authenticity of the false sample G(z) based on the real sample x. The ideal generative network G has a strong ability to forge, and the false sample G(z) produced makes it difficult for the discriminative network D to identify it. On the contrary, the ideal discriminative network D should have a strong ability to detect falsehoods and accurately determine the authenticity of the false sample G(z).

[0035] The present disclosure utilizes the game characteristics of the generating network G and the discriminant network D. In the point cloud edge detection business, the generative adversarial network is iteratively trained, so that the generating network G has a good counterfeiting ability. The discriminant network D is based on real samples (real samples are "real labeled point clouds", and the so-called "real labeled point clouds" are point cloud labeled data that are considered to be real during the training process). It is difficult to identify the authenticity of false samples (false samples are "generated labeled point clouds", and the so-called "generated labeled point clouds" are point cloud labeled data generated by the generating network during the training process to be identified as real) produced by the generating network G, which means that the accuracy of the edge detection result of the generating network G at this time is very high. Then, using the generating network G in the trained generating adversarial network, the edge detection of the point cloud to be detected can be performed, and the accuracy of the edge detection result is very high.

[0036] Combine the following Figure 1 The point cloud edge detection solution based on the generative adversarial network disclosed in the present invention is specifically described.

[0037] Figure 1 FIG. 1 is a flow chart of a point cloud edge detection method according to some embodiments of the present disclosure. Figure 1 As shown, the method of this embodiment includes:

[0038] In step 110, the generative adversarial network is trained using point cloud training data. The generative adversarial network includes a generative network and a discriminative network. The point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud.

[0039] The training process includes: the generation network performs edge detection on the first input point cloud to obtain a generated labeled point cloud, the generated labeled point cloud is input into the discriminant network for authenticity judgment, the discriminant network judges the authenticity of the generated labeled point cloud based on the real labeled point cloud, and through iterative training of the generative adversarial network, the difference between the true probability and the false probability in the judgment result of the discriminant network is made smaller than the preset value, which means that the discriminant network D is difficult to identify the authenticity of the false samples (i.e., "generated labeled point cloud") produced by the generation network G based on the real samples (i.e., "real labeled point cloud"), that is to say, the generation network G has a good counterfeiting ability, and the accuracy of its edge detection results is very high.

[0040] For example, a "real annotated point cloud" can be obtained by manual annotation. Different edge data can be annotated for different applications. Robustness can be improved by enriching training samples.

[0041] It should be noted that the descriptions such as “first” and “second” in the present disclosure are only used to distinguish different objects and are not used to indicate meanings such as size or time sequence.

[0042] In step 120, edge detection is performed on the second input point cloud to be detected using the generative network in the trained generative adversarial network.

[0043] In some embodiments, the second input point cloud is obtained by thinning the original point cloud to be detected. The thinning algorithm may, for example, use a k-means algorithm to select representative partial points from each of the clustered categories. Thus, the number of data points is minimized while ensuring that the shape of the vector curve remains unchanged.

[0044] In some embodiments, binary edge detection may be performed, that is, detecting whether a point in the point cloud is an edge point, and the corresponding edge detection result is the probability of whether each point of the second input point cloud is an edge point.

[0045] In some embodiments, semantic edge detection may be performed, that is, detecting the semantic category of the edge of the point cloud, and the corresponding edge detection result is the probability of the semantic category of each point of the second input point cloud. For example, the semantic category corresponding to the point cloud of the street scene includes, for example, curbs, lane lines, etc.

[0046] In some embodiments, the generative adversarial network may be, for example, a generative adversarial network built based on a deep neural network.

[0047] The present disclosure also proposes a multi-scale generative adversarial network, whose convolution layer is provided with multiple convolution branches of different scales, and each convolution branch performs convolution operations independently, so as to better preserve details and improve accuracy in point cloud edge detection.

[0048] In some embodiments, the generative network and the discriminative network in the generative adversarial network have similar structures. For example, both the generative network and the discriminative network include a convolutional layer (with multiple convolutional branches of different scales), a cascade layer, a fully connected layer, a normalization layer, and an output layer. In addition, the discriminative network also includes an average layer. The functions of the fully connected layer and the normalization layer can be referred to in the prior art, and will not be repeated here.

[0049] In some embodiments, the convolutional layers of the generative network and the discriminative network are provided with multiple convolutional branches of different scales; the cascaded layers of the generative network and the discriminative network keep the point cloud position unchanged and connect multiple convolutional branches in series. Thus, the convolutional layers of the generative network and the discriminative network downsample the point cloud input to the convolutional layer into point clouds of multiple granularities, and the point clouds of each granularity are independently convoluted, and the results of each operation are cascaded into a point cloud.

[0050] Among them, the parameter N of the first convolution layer located in different convolution branches is different, and the parameter N of the last convolution layer located in different convolution branches is the same, wherein the parameter N represents the number of points in the output point cloud.

[0051] Among them, the cascade layer keeps the parameter N of the last convolution layer in each convolution branch unchanged, and superimposes the parameter C of the last convolution layer in each convolution branch, where the parameter C represents the number of feature channels of the points in the output point cloud.

[0052] like Figure 2 The generative network shown in the figure performs edge detection on the input point cloud, where the dotted box is the data layer and the solid box is the network layer. "8192×4" represents the input point cloud, that is, the original point cloud with 8192 input point clouds and 4 channels (such as color information: red (R), green (G), blue (B) and reflection intensity (Intensity)). "8192×#cls" represents the output layer, which outputs the edge detection result of the input point cloud. If it is a binary edge detection (whether a point in the point cloud is an edge point or not), the task "#cls" = 2. If it is a semantic edge detection, the task "#cls" is the number of semantic categories. In the structure of the network layer, "N = ..., C = ..." represents the convolution layer (X-Conv layer), which convolves the input point cloud into the output point cloud, where the parameter N represents the number of points in the output point cloud, and the parameter C represents the number of feature channels of the points in the output point cloud. The number of neighborhood points K (i.e., the number of neighborhood points in the input point cloud that participate in the calculation of each output point cloud point) is set to, for example, 8, and the expansion step D is, for example, 4. The values ​​of K and D are not shown in the figure. The calculation process of the X-Conv operation can refer to the prior art and will not be repeated here. "concat" represents a cascade layer, which is used to keep the point cloud position unchanged and connect multiple convolution branches in series. Figure 24 convolution branches are shown as an example, so that the output of "N = 8192, C = 256" is downsampled into 4 point clouds of different granularities "N = 512, C = 512", "N = 1024, C = 512", "N = 2048, C = 512", "N = 4096, C = 512", and then the four point clouds are independently convolved. Finally, the cascade layer keeps the parameter N of the convolution layer of the last layer in each convolution branch unchanged, and superimposes the parameter C of the convolution layer of the last layer in each convolution branch, so that the 4 point clouds output by "N = 4096, C = 64" are connected in series into one point cloud "4096 × 256", and subsequent operations are performed. "FC(x)" represents a fully connected layer, where x is the number of output channels. "softmax" represents a normalization layer, and the Softmax formula is: It means to normalize the features of the output channel xi, and exp means the exponential operation.

[0053] like Figure 3 The discriminant network shown in the figure discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud. The dotted box is the data layer, and the solid box is the network layer. "8192×5" represents the point cloud to be judged, that is, the "generated annotated point cloud" output by the generating network. The number of points in the point cloud to be judged is 8192 and the number of channels is 5 (such as color information: red (R), green (G), blue (B), reflection intensity (Intensity), and task cls). "1×2" represents the output layer, which outputs the probability that the point cloud to be judged "generated annotated point cloud" is true and false. The functions of the X-Conv layer, "concat" layer, "softmax" layer, and "FC(x)" layer in the network layer structure are the same as those of the generating network. The function of the "average" layer is to calculate the average value of the point cloud channel, for example, to calculate the average value of 256 points in each channel.

[0054] It should be noted that Figure 2 and Figure 3 The specific values ​​in, for example, the number of point cloud points is 8192, the value of parameter x in parameter N and parameter C, FC(x) is only an example and can be set to other values ​​according to business needs.

[0055] Taking street view laser point cloud edge detection as an example, it includes training phase and testing phase.

[0056] Training phase: First, a large amount of color point cloud data is captured using existing public data sets or vehicle-mounted laser camera devices, and point cloud slices are performed along the driving trajectory. For example, a point cloud with a radius of 20 meters centered on the current vehicle position is obtained every 10 meters; then, each point cloud is processed as follows: a) 8192 point cloud points are extracted using the K-Mean method, b) the edge attributes of the point cloud are annotated by manually drawing bounding boxes, and c) the generative adversarial network is iteratively trained using the training data until the generative network can produce false samples that are difficult for the discriminative network to identify as true or false - "generate annotated point cloud".

[0057] Testing phase: First, the vehicle-mounted laser camera device collects and captures a large amount of color point cloud data, and slices the point cloud along the driving trajectory, for example, obtaining a point cloud with a radius of 20 meters centered on the current vehicle position every 10 meters; then, each point cloud is processed as follows ab), namely a) using the K-Mean method to extract, for example, 8192 point cloud points, and b) using the generative network for edge detection (i.e. edge annotation). In addition, the point cloud slices after edge annotation can also be spliced.

[0058] In some embodiments, the point cloud edge detection disclosed in the present invention can be further applied to business scenarios such as point cloud compression, simulation construction, and environmental perception. In point cloud compression, non-edge points are eliminated and edge points that fall on the contour of the object are retained, which can reduce the number of point clouds. In simulation construction, vectorization of edge points can build a simple simulation environment. In environmental perception, semanticization of edge points helps to perceive the environment. For example, elements such as lane lines and curbs in street scenes often fall on the edges, and semanticization of edge points helps the intelligent analysis of vehicle-mounted systems.

[0059] Figure 4 Schematic diagram of the structure of the point cloud edge detection device in some embodiments of the present disclosure. Figure 4 As shown, the point cloud edge detection device 400 of this embodiment includes:

[0060] Memory 410; and

[0061] The processor 420 is coupled to the memory, and the processor is configured to execute the point cloud edge detection method of any one of the aforementioned embodiments based on the instructions stored in the memory.

[0062] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0063] Figure 5 Schematic diagram of the structure of the point cloud edge detection device in some embodiments of the present disclosure. Figure 5 As shown, the point cloud edge detection device 500 of this embodiment includes:

[0064] The training unit 510 is configured to train a generative adversarial network using point cloud training data, wherein the generative adversarial network includes a generative network and a discriminative network, and the point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, wherein the training process includes: the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, and the discriminative network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, and through iterative training of the generative adversarial network, the difference between the true probability and the false probability in the discrimination result of the discriminative network is made smaller than a preset value.

[0065] The detection unit 520 is configured to use the generative network in the trained generative adversarial network to perform edge detection on the second input point cloud to be detected.

[0066] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the point cloud edge detection method of any of the aforementioned embodiments.

[0067] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A point cloud edge detection method, comprising: The generative adversarial network is trained using point cloud training data, wherein the generative adversarial network includes a generative network and a discriminative network, and the point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, wherein the training process includes: the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, and the discriminative network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, and through iterative training of the generative adversarial network, the difference between the true probability and the false probability in the discrimination result of the discriminative network is made less than a preset value, wherein the generative network and the discriminative network include a convolution layer and a cascade layer, the convolution layer is provided with a plurality of convolution branches of different scales, the cascade layer keeps the position of the point cloud unchanged and connects a plurality of convolution branches in series, the parameter N of the convolution layer of the last layer in different convolution branches is the same, the cascade layer keeps the parameter N of the convolution layer of the last layer in each convolution branch unchanged, and superimposes the parameter C of the convolution layer of the last layer in each convolution branch, wherein the parameter N represents the number of points of the output point cloud, and the parameter C represents the number of feature channels of the points of the output point cloud; The generative network in the trained generative adversarial network is used to perform edge detection on the second input point cloud to be detected.

2. The method of claim 1, wherein: The convolutional layers of the generative network and the discriminative network downsample the point cloud input to the convolutional layer into point clouds of multiple granularities. The point clouds of each granularity are convolved independently, and the results of each operation are cascaded into a point cloud.

3. The method of claim 1, wherein: The parameter N of the first convolution layer in different convolution branches is different.

4. The method of claim 1, wherein: The results of edge detection include: The probability of whether each point of the second input point cloud is an edge point; or, The probability of the semantic category of each point in the second input point cloud.

5. The method of claim 1, wherein: The second input point cloud is obtained by thinning the original point cloud to be detected.

6. A point cloud edge detection device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the point cloud edge detection method according to any one of claims 1 to 5 based on instructions stored in the memory.

7. A point cloud edge detection device, comprising: The training unit is configured to train a generative adversarial network using point cloud training data, wherein the generative adversarial network includes a generative network and a discriminative network, and the point cloud training data includes a first input point cloud for training and a real annotated point cloud obtained by annotating the first input point cloud, wherein the training process includes: the generative network performs edge detection on the first input point cloud to obtain a generated annotated point cloud, the discriminative network discriminates the authenticity of the generated annotated point cloud based on the real annotated point cloud, and through iterative training of the generative adversarial network, the true probability and the false probability in the discrimination result of the discriminative network are increased. The difference between them is less than a preset value, wherein the generating network and the discriminating network include a convolution layer and a cascade layer, the convolution layer is provided with a plurality of convolution branches of different scales, the cascade layer keeps the point cloud position unchanged and connects a plurality of convolution branches in series, the parameter N of the convolution layer of the last layer in different convolution branches is the same, the cascade layer keeps the parameter N of the convolution layer of the last layer in each convolution branch unchanged, and superimposes the parameter C of the convolution layer of the last layer in each convolution branch, wherein the parameter N represents the number of points of the output point cloud, and the parameter C represents the number of feature channels of the points of the output point cloud; The detection unit is configured to use the generative network in the trained generative adversarial network to perform edge detection on the second input point cloud to be detected.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the point cloud edge detection method according to any one of claims 1 to 5.

Images