A laser point cloud detection and segmentation method and system

Abstract

The application discloses a kind of laser point cloud detection segmentation method and system.The laser point cloud detection segmentation method includes using convolutional neural network as feature extractor, the semantic feature extraction and edge feature extraction of detection image are carried out by the feature extractor, obtain semantic feature and edge feature;After the convolutional neural network, two network branches in parallel are used as object detection branch and semantic segmentation branch, the semantic feature and the edge feature are processed by object detection branch and semantic segmentation branch, wherein the object detection branch processes object detection task, and the semantic segmentation branch processes semantic segmentation task;Loss function is used to balance processing the object detection task and the semantic segmentation task to optimize iteration.The application uses convolutional neural network to extract basic features, uses two branches to process different tasks respectively, and uses loss function to balance different tasks to make them quickly optimize iteration.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology in general scenarios, and in particular to a laser point cloud detection and segmentation method and system in scenarios with dense pedestrians and vehicles. Background Technology

[0002] Common laser point cloud processing methods typically use 3D point clouds as network input or represent point clouds as voxels. However, both methods share the drawback of being computationally intensive, making them difficult to run in real time on automotive edge devices.

[0003] Current methods for 3D target detection using laser point clouds fall into three categories: those based on raw point clouds, those based on voxel representation, and those based on top-view projection. Methods using raw point clouds suffer from large input data volumes, making them unsuitable for practical deployment by in-vehicle edge devices. Voxel representation-based methods, on the other hand, present a trade-off between accuracy and memory usage, preventing edge devices from achieving accurate results even with smooth operation.

[0004] Detection and segmentation are two distinct tasks, typically requiring different neural networks. Currently, the size and accuracy of neural network models are strongly correlated, making it difficult for edge devices to simultaneously run two deep neural networks and achieve good results in both. Using a single network model, however, presents cross-task domain fusion problems, making it difficult to handle two different tasks with a single network structure.

[0005] Due to the different mechanisms of cameras and LiDAR, multimodal fusion faces significant challenges. In addition, the limited computing power of onboard processors makes the need to use a single deep learning model for multitasking increasingly urgent.

[0006] Therefore, there is an urgent need to study a laser point cloud detection and segmentation system to enable multi-task processing using a single network structure and to run in real time on vehicle-mounted edge devices. Summary of the Invention

[0007] This invention proposes a laser point cloud detection and segmentation method and system to overcome at least one technical problem existing in the prior art.

[0008] According to a first aspect of the embodiments of this specification, a laser point cloud detection and segmentation method is provided, comprising: employing a convolutional neural network as a feature extractor, performing semantic feature extraction and edge feature extraction on a detection image through the feature extractor to obtain semantic features and edge features; after the convolutional neural network, using two parallel network branches as an object detection branch and a semantic segmentation branch, processing the semantic features and the edge features through the object detection branch and the semantic segmentation branch, wherein the object detection branch processes the object detection task and the semantic segmentation branch processes the semantic segmentation task; and employing a loss function to balance the object detection task and the semantic segmentation task for optimization iteration.

[0009] Optionally, the step of using a convolutional neural network as a feature extractor to extract semantic features and edge features from the detected image includes: constructing a convolutional neural network using separable convolution; extracting semantic features by performing a nonlinear transformation on the projected two-dimensional image using the convolutional neural network; and extracting edge features from the two-dimensional image using a skip connection method to obtain a feature map of the two-dimensional image. The semantic features include relevant information or contextual information about the detected object and its background, the edge features include the edge contours of the detected object, and the skip connection method includes pixel-by-pixel addition of a feature map of a preset size obtained through upsampling with a shallow feature map of the corresponding size for feature fusion.

[0010] Optionally, the semantic segmentation branch processes the semantic segmentation task, including: restoring the feature map obtained by the convolutional neural network to the original image size by upsampling, using the number of channels of the last layer of the convolutional neural network as the number of categories of the detected object; processing the feature map restored to the original image size using the Softmax function to obtain the probability of each pixel of the feature map belonging to each category, and determining the category to which each pixel belongs based on the probability.

[0011] Optionally, the Softmax function expression (1) is:

[0012]

[0013] Where z i Let be the pixel value of the i-th channel, and C be the number of categories;

[0014] The output value of each semantic segmentation branch is compared with the corresponding pre-given ground truth label value, and the network parameters of the semantic segmentation branch are iteratively updated. The iterative update of the network parameters of the semantic segmentation branch is based on the loss function L. point As shown in equation (2):

[0015] L point =-α(1-p i ) γ log(p i (2)

[0016] Where p i For the final output of the split branch, α is the balance factor and γ is the modulation factor.

[0017] Optionally, the object detection branch includes a first sub-branch, a second sub-branch, and a third sub-branch. The first sub-branch is used to classify the detected object, the second sub-branch is used to predict the target bounding box for calculating depth information, and the third sub-branch is used to estimate the orientation of the detected object. The object detection branch processes the object detection task, including: the first sub-branch determining the category of the detected object using a Softmax function; the second sub-branch using a regression method to add a Gaussian kernel to the feature map of the two-dimensional image to obtain a two-dimensional response distribution and a heatmap, taking the highest point in the two-dimensional response distribution as the object center point, and obtaining the target bounding box based on the distance from the object center point and the predicted center point to the object bounding box; and the third sub-branch estimating the orientation of the detected object by calculating the depth information of the feature map of the two-dimensional image and comparing it with the depth information in the target bounding box.

[0018] Optionally, the object detection branch, which processes the object detection task, further includes: the object detection branch employs a loss function to characterize the difference between the output value of the object detection branch and the corresponding pre-given real labeled value.

[0019] Optionally, the loss function used in the object detection branch includes:

[0020] The loss function L reg (x,r,g),L cls (x,p) are shown in equations (3) and (4) below:

[0021]

[0022]

[0023] Where M∈{x0,y 0, The numbers {z0, w, h, l, θ} represent the seven values ​​that need to be regressed for a detected object, which are the center coordinates (x0, y ... 0, (z0), length, width, height (w, h, l), and deviation angle θ; Whether the i-th pixel in the feature map of the two-dimensional image corresponds to the ground truth bounding box of a detected object; and These represent the output value of the second sub-branch corresponding to the regression value and the pre-given true labeled value, respectively; and These represent the probability outputs for positive and negative samples, respectively.

[0024] Optionally, the step of balancing the object detection task and the semantic segmentation task using a loss function to optimize the iteration includes: integrating the loss of the object detection branch and the loss of the semantic segmentation branch in a linear weighted manner; and using different weight coefficients before the losses of different tasks, wherein the weight coefficients are composed of the reciprocal of the cumulative time of the squared gradients of different tasks.

[0025] Optionally, the semantic segmentation task's features assist the object detection branch, and the step of using a loss function to balance the object detection task and the semantic segmentation task to optimize the iteration further includes:

[0026] We first train the semantic segmentation branch using a loss function, and then train the object detection branch.

[0027] The loss function Loss, which balances the object detection and semantic segmentation tasks, is shown in equation (5) below:

[0028]

[0029] Where σ represents the adjustment factor of the loss function in the semantic segmentation branch, and ω represents the adjustment factor of the regression loss in the object detection branch. This represents the adjustment factor for the classification loss in the object detection branch.

[0030] According to a second aspect of the embodiments of this specification, a laser point cloud detection and segmentation system is provided, including a feature extraction module, a task processing module, and a rate balancing module. The feature extraction module is configured to use a convolutional neural network as a feature extractor to extract semantic features and edge features from a detection image, obtaining semantic features and edge features. The task processing module is configured to use two parallel network branches after the convolutional neural network as an object detection branch and a semantic segmentation branch, processing the semantic features and edge features through the object detection branch and the semantic segmentation branch, wherein the object detection branch handles object detection tasks, and the semantic segmentation branch handles semantic segmentation tasks. The rate balancing module is configured to use a loss function to balance the object detection task and the semantic segmentation task for optimization iteration.

[0031] According to a third aspect of the embodiments of this specification, a computing device is provided, including a storage device and a processor, the storage device being used to store a computer program, and the processor running the computer program to cause the computing device to perform the steps of the laser point cloud detection and segmentation method.

[0032] According to a fourth aspect of the embodiments of this specification, a storage medium is provided that stores a computer program used in the computing device, which, when executed by a processor, implements the steps of the laser point cloud detection and segmentation method.

[0033] The technical effects of this invention are as follows:

[0034] This specification provides a laser point cloud detection and segmentation method and system. The method uses a convolutional neural network to extract features from the projected two-dimensional image of the laser point cloud, obtaining semantic and edge features. Two parallel network branches are then used to handle object detection and semantic segmentation tasks respectively. A loss function is used to balance the processing speed of different tasks, enabling rapid optimization and iteration of the network. The method in this embodiment, based on a convolutional neural network, unifies detection and segmentation tasks through parallel network branches, using a single neural network. This satisfies the requirements of in-vehicle edge devices and achieves both segmentation and detection objectives. Furthermore, the loss function balances different tasks for rapid optimization and iteration. In this embodiment, processing the projected front view reduces model complexity and computational load, allowing for faster deployment and operation on in-vehicle edge devices. It offers high real-time performance, solving the problem of high computational demands and difficulty in meeting real-time requirements in autonomous driving scenarios. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a laser point cloud detection and segmentation method according to an embodiment of this specification.

[0037] Figure 2 This is a schematic diagram of the neural network structure in an embodiment of this specification;

[0038] Figure 3 This is a schematic diagram of the structure of a laser point cloud detection and segmentation system according to an embodiment of this specification;

[0039] Figure 4This is a schematic diagram of the structure of the computing device according to an embodiment of this specification;

[0040] Figure 5 This is a schematic diagram of the structure of the storage medium in an embodiment of this specification. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0043] This specification discloses a laser point cloud detection and segmentation method and system, which will be described in detail below.

[0044] Figure 1 This is a flowchart illustrating a laser point cloud detection and segmentation method according to an embodiment of this specification. Figure 1 As shown in the embodiments of this specification, a laser point cloud detection and segmentation method is provided, including:

[0045] S110. A convolutional neural network is used as a feature extractor. The feature extractor is used to extract semantic features and edge features from the detected image to obtain semantic features and edge features.

[0046] In a specific embodiment, step S110, which uses a convolutional neural network as a feature extractor to extract semantic features and edge features from the detected image, and obtains semantic features and edge features, includes:

[0047] S112. Construct a convolutional neural network using separable convolution.

[0048] S114. The semantic features are extracted by nonlinear transformation of the projected two-dimensional image through the convolutional neural network, and the edge features of the two-dimensional image are extracted by using the skip connection method to obtain the feature map of the two-dimensional image. The semantic features include relevant information or contextual information of the detected object and its background, the edge features include the edge contour of the detected object, and the skip connection method includes adding the feature map of the preset size obtained by upsampling to the shallow feature map of the corresponding size pixel by pixel to perform feature fusion.

[0049] Semantic features refer to abstract features in an image, usually referring to information related to the object and the background or contextual information. Semantic features are helpful for classifying objects in an image. Edge features refer to the edge contours of objects, which are helpful for locating objects. The two types of features respectively meet the needs of object detection and semantic segmentation.

[0050] Considering the limitations of computing power and storage space of in-vehicle edge devices, depthwise separable convolution is selected as the basic module to construct convolutional neural networks. The convolutional neural network is used as a feature extractor to extract semantic and edge features from the projected front view, i.e., the two-dimensional image.

[0051] To better utilize edge features to optimize segmentation results, in one specific embodiment, a skip connection method is used to add the feature map, which has been upsampled and enlarged by a factor of 2, to the shallow feature map of the corresponding size pixel by pixel to achieve feature fusion.

[0052] After processing by the convolutional neural network in S110, the resulting feature map contains rich semantic features and shallow edge features. Two branches are built after the convolutional neural network to handle object detection and semantic segmentation tasks respectively.

[0053] S120. After the convolutional neural network, two parallel network branches are used as an object detection branch and a semantic segmentation branch. The semantic features and the edge features are processed through the object detection branch and the semantic segmentation branch. The object detection branch processes the object detection task, and the semantic segmentation branch processes the semantic segmentation task.

[0054] The semantic segmentation branches adopt a cylindrical structure.

[0055] In a specific embodiment, the semantic segmentation branch processes the semantic segmentation task, including:

[0056] S1222. The feature map obtained by the convolutional neural network is restored to the original image size by upsampling, and the number of channels of the last layer of the convolutional neural network is used as the number of categories of the detected object.

[0057] Upsampling restores the feature map obtained from the convolutional neural network to the original image size, and the number of channels in the last layer is equal to the number of object categories. The Softmax function can be used to obtain the probability of each pixel belonging to a certain category. The category with the highest probability is the category to which the pixel belongs. Classifying each pixel into categories completes the semantic segmentation task.

[0058] S1224. The Softmax function is used to process the feature map restored to the original image size to obtain the probability of each pixel in the feature map belonging to each category, and the category to which each pixel belongs is determined according to the probability.

[0059] In one specific implementation, the Softmax function expression (1) is:

[0060]

[0061] Where z i Let be the pixel value of the i-th channel, and C be the number of categories.

[0062] The Softmax function can be used to convert the output values ​​of multi-class classification into a probability distribution in the range [0,1].

[0063] During neural network training, the network parameters are iteratively updated by comparing each output value with the true and correct label value. The updates are based on a loss function, which is minimized to make the network output closer to the true label.

[0064] The output value of each semantic segmentation branch is compared with the corresponding pre-given ground truth label value, and the network parameters of the semantic segmentation branch are iteratively updated. The iterative update of the network parameters of the semantic segmentation branch is based on the loss function L. point As shown in equation (2):

[0065] L point =-α(1-p i ) γ log(p i (2)

[0066] Where p i For the final output of the split branch, α is the balance factor and γ is the modulation factor.

[0067] In the specific implementation method, p in the formula iThe final output of the segmentation branch represents the probability that each pixel belongs to a certain category. α is a balancing factor used to balance the uneven ratio of positive and negative samples. Usually, α = 0.25 is chosen. γ is a modulation factor, which aims to reduce the weight of easily distinguishable samples and make the model pay more attention to samples that are difficult to distinguish. γ = 2 can be chosen.

[0068] In a specific embodiment, the object detection branch includes a first sub-branch, a second sub-branch, and a third sub-branch. The first sub-branch is used to classify the detected object, the second sub-branch is used to predict the bounding box for calculating depth information, and the third sub-branch is used to estimate the orientation of the detected object. The object detection branch handles the object detection task, including:

[0069] S1242, The first sub-branch determines the category of the detected object using the Softmax function.

[0070] The first sub-branch is the object classification branch, which is similar to the classification branch in semantic segmentation tasks. The softmax function is used to determine the category of the detected object.

[0071] S1244. The second sub-branch uses a regression method to add a Gaussian kernel to the feature map of the two-dimensional image to obtain a two-dimensional response distribution and a heat map. The highest point in the two-dimensional response distribution is taken as the object center point, and the target box is obtained based on the distance from the object center point and the predicted center point to the object bounding box.

[0072] The bounding box prediction uses a regression approach. A Gaussian kernel is added to the feature map to obtain a two-dimensional response distribution, which is then used to obtain a heatmap. The highest response point is the object center point. The object bounding box is obtained based on the object center point and the distance from the predicted center point to the object bounding box.

[0073] S1246. The third sub-branch estimates the orientation of the detected object by calculating the depth information of the feature map of the two-dimensional image and comparing it with the depth information in the target box.

[0074] The object orientation is estimated. Specifically, since different orientations of an object present different depth information in the detected target box, the object orientation is estimated by comparing the depth information in the detected target box.

[0075] Similar to semantic segmentation tasks, object detection tasks also require a loss function to characterize the difference between the network output and the true labeled values.

[0076] In a specific embodiment, the object detection branch, which processes the object detection task, further includes:

[0077] S1248. The object detection branch uses a loss function to characterize the difference between the output value of the object detection branch and the corresponding pre-given real label value.

[0078] In one specific implementation, the loss function used by the object detection branch includes:

[0079] The loss function L reg (x,r,g),L cls (x,p) are shown in equations (3) and (4) below:

[0080]

[0081]

[0082] Where M∈{x0,y 0, The numbers {z0, w, h, l, θ} represent the seven values ​​that need to be regressed for a detected object, which are the center coordinates (x0, y ... 0, (z0), length, width, height (w, h, l), and deviation angle θ; Whether the i-th pixel in the feature map of the two-dimensional image corresponds to the ground truth bounding box of a detected object; and These represent the output value of the second sub-branch corresponding to the regression value and the pre-given true labeled value, respectively; and These represent the probability outputs for positive and negative samples, respectively.

[0083] In multi-task learning, convolutional neural networks have multiple network branches, each corresponding to a different loss function. If these loss functions are simply summed, as the learning process progresses, the gradient of one loss function that decreases rapidly will negatively impact the overall learning process of the network.

[0084] S130. The object detection task and the semantic segmentation task are balanced using a loss function to optimize the iteration.

[0085] In a specific embodiment, step S130, which involves balancing the object detection task and the semantic segmentation task using a loss function to optimize the iteration, includes:

[0086] S132. The loss of the object detection branch and the loss of the semantic segmentation branch are integrated by linear weighting.

[0087] S134. Different weighting coefficients are used before the loss for different tasks, where each weighting coefficient is composed of the inverse of the cumulative time of the squared gradients for different tasks. This allows tasks that update faster to use a smaller learning rate, thereby balancing the overall learning process of the network.

[0088] Since semantic segmentation features can assist in the detection branch, the segmentation branch is trained first for a period of time before the detection branch is trained, thus achieving the goal of using segmentation features to assist in the detection task.

[0089] In a specific embodiment, the feature-assisted object detection branch of the semantic segmentation task, and the step of S130, balancing the object detection task and the semantic segmentation task using a loss function to optimize the iteration, further include:

[0090] S136. Use a loss function to first train the semantic segmentation branch and then train the object detection branch.

[0091] The loss function Loss, which balances the object detection and semantic segmentation tasks, is shown in equation (5) below:

[0092]

[0093] Where σ represents the adjustment factor of the loss function in the semantic segmentation branch, and ω represents the adjustment factor of the regression loss in the object detection branch. This represents the adjustment factor for the classification loss in the object detection branch.

[0094] The proportion of loss between segmentation and detection branches varies under different task conditions. The adjustment factors of each loss function can be adjusted according to the actual situation to balance the training of the network.

[0095] Figure 2 This is a schematic diagram of the neural network structure of an embodiment of this specification. For example... Figure 2 As shown, the neural network in this embodiment consists of a backbone network 210, an object detection branch 220, and a semantic segmentation branch 230. The backbone network 210 is composed of several layers of convolutional neural networks and an upsampling network. The object detection branch 220 consists of an upsampling network and branches of three convolutional neural networks. The semantic segmentation branch 230 adopts a cylindrical structure. In this embodiment, three-dimensional point cloud data is converted into a two-dimensional image through front view projection, and the conversion process is recorded. A high-efficiency convolutional neural network (backbone network 210) is used to extract semantic and edge features from the two-dimensional image. Then, two parallel network branches (object detection branch 220 and semantic segmentation branch 230) handle the segmentation and detection tasks respectively. Finally, the detection and segmentation results of the three-dimensional point cloud are obtained through the correspondence between point clouds and pixels during the conversion process.

[0096] Figure 3 This is a schematic diagram of the structure of a laser point cloud detection and segmentation system according to an embodiment of this specification. Figure 3 As shown in the embodiments of this specification, a laser point cloud detection and segmentation system 300 is provided, including a feature extraction module 310, a task processing module 320, and a rate balancing module 330, wherein...

[0097] The feature extraction module 310 is configured to use a convolutional neural network as a feature extractor to extract semantic features and edge features from the detected image, thereby obtaining semantic features and edge features.

[0098] The task processing module 320 is configured to use two parallel network branches as an object detection branch and a semantic segmentation branch after the convolutional neural network, and to process the semantic features and the edge features through the object detection branch and the semantic segmentation branch, wherein the object detection branch processes the object detection task and the semantic segmentation branch processes the semantic segmentation task.

[0099] The rate balancing module 330 is configured to use a loss function to balance the object detection task and the semantic segmentation task for optimization iteration.

[0100] Figure 4 This is a schematic diagram of the structure of a computing device according to an embodiment of this specification. Figure 4 As shown in the embodiment of this specification, a computing device 400 is provided, including a storage device 410 and a processor 420. The storage device 410 is used to store a computer program, and the processor 420 runs the computer program to enable the computing device to perform the steps of the laser point cloud detection and segmentation method.

[0101] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of this specification. For example... Figure 5 As shown in the embodiment of this specification, a storage medium 500 is provided, which stores a computer program 510 used in the computing device. When the computer program 510 is executed by a processor, it implements the steps of the laser point cloud detection and segmentation method.

[0102] In summary, this specification provides a laser point cloud detection and segmentation method and system. It extracts features from the projected two-dimensional image of the laser point cloud to obtain semantic and edge features. Two parallel network branches are then used to handle object detection and semantic segmentation tasks respectively. A loss function is used to balance the processing speed of different tasks, enabling rapid network optimization and iteration. Based on a convolutional neural network, the detection and segmentation tasks are unified through parallel network branches, using a single neural network. This satisfies the requirements of in-vehicle edge devices and achieves both segmentation and detection. The loss function balances different tasks for rapid optimization and iteration. Processing the projected view reduces model complexity and computational load, allowing for faster deployment and operation on in-vehicle edge devices with high real-time performance. This solves the problem of high computational demands and difficulty in meeting real-time requirements in autonomous driving scenarios.

[0103] The above system and device embodiments correspond to the method embodiments and have the same technical effects. For detailed descriptions, please refer to the method embodiments. The device embodiments are derived based on the method embodiments; detailed descriptions can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0104] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser point cloud detection and segmentation method, characterized in that, include: A convolutional neural network is used as a feature extractor. The feature extractor is used to extract semantic features and edge features from the detected image to obtain semantic features and edge features. Following the convolutional neural network, two parallel network branches are used as an object detection branch and a semantic segmentation branch. The semantic features and edge features are processed through the object detection branch and the semantic segmentation branch, wherein the object detection branch handles the object detection task and the semantic segmentation branch handles the semantic segmentation task. A loss function is used to balance the object detection task and the semantic segmentation task for optimization iteration; The object detection branch includes a first sub-branch, a second sub-branch, and a third sub-branch. The first sub-branch is used to classify the detected object, the second sub-branch is used to predict the bounding box for calculating depth information, and the third sub-branch is used to estimate the orientation of the detected object. The object detection branch handles the object detection task, including: The first sub-branch determines the category of the detected object using the Softmax function; The second sub-branch uses a regression approach to add a Gaussian kernel to the feature map of the two-dimensional image to obtain a two-dimensional response distribution and a heatmap. The highest point in the two-dimensional response distribution is taken as the object center point, and the target box is obtained based on the distance from the object center point and the predicted center point to the object bounding box. The third sub-branch estimates the orientation of the detected object by calculating the depth information of the feature map of the two-dimensional image and comparing it with the depth information in the target box. The loss function used in the object detection branch includes: The loss function L reg (x,r,g),L cls (x,p) are shown in equations (3) and (4) below: Where M∈{x0,y0,z0,w,h,l,θ} represents the 7 values ​​that need to be regressed for a detected object, namely the center coordinates (x0,y0,z0), length, width and height (w,h,l) and skew angle θ of the target box; Whether the i-th pixel in the feature map of the two-dimensional image corresponds to a ground truth bounding box of a detected object; and These represent the output value of the second sub-branch corresponding to the regression value and the pre-given true labeled value, respectively; and These represent the probability outputs for positive and negative samples, respectively.

2. The method according to claim 1, characterized in that, The step of using a convolutional neural network as a feature extractor to extract semantic and edge features from the detected image, thereby obtaining semantic and edge features, includes: Convolutional neural networks are constructed using separable convolutions; The convolutional neural network is used to perform a nonlinear transformation on the projected 2D image to extract semantic features, and a skip connection method is used to extract edge features from the 2D image to obtain the feature map of the 2D image. The semantic features include information or contextual information about the detected object and its background, the edge features include the edge contour of the detected object, and the skip connection method includes pixel-by-pixel addition of a feature map of a preset size obtained through upsampling with a shallow feature map of the corresponding size for feature fusion.

3. The method according to claim 1, characterized in that, The semantic segmentation branch processes the semantic segmentation task, including: The feature map obtained by the convolutional neural network is restored to the original image size by upsampling, and the number of channels of the last layer of the convolutional neural network is used as the number of categories of the detected object. The Softmax function is used to process the feature map restored to the original image size to obtain the probability that each pixel of the feature map belongs to each category, and the category to which each pixel belongs is determined based on the probability.

4. The method according to claim 3, characterized in that, The expression for the Softmax function (1) is: Where z i Let be the pixel value of the i-th channel, and C be the number of categories; The output value of each semantic segmentation branch is compared with the corresponding pre-given ground truth label value, and the network parameters of the semantic segmentation branch are iteratively updated. The iterative update of the network parameters of the semantic segmentation branch is based on the loss function L. point As shown in equation (2): L point =-α(1-p i ) γ log(p i ) (2) Where p i For the final output of the split branch, α is the balance factor and γ is the modulation factor.

5. The method according to claim 1, characterized in that, The object detection branch, which handles object detection tasks, also includes: The object detection branch uses a loss function to characterize the difference between the output value of the object detection branch and the corresponding pre-given true label value.

6. The method according to claim 1, characterized in that, The step of balancing the object detection task and the semantic segmentation task using a loss function to optimize the iteration includes: The loss of the object detection branch and the loss of the semantic segmentation branch are integrated by linear weighting. Different weighting coefficients are used before the loss for different tasks, where each weighting coefficient is composed of the reciprocal of the time cumulative sum of the squared gradients for different tasks.

7. The method according to claim 1, characterized in that, The feature-assisted object detection branch of the semantic segmentation task, and the step of balancing the object detection task and the semantic segmentation task using a loss function to optimize the iteration, further include: We first train the semantic segmentation branch using a loss function, and then train the object detection branch. The loss function Loss, which balances the object detection and semantic segmentation tasks, is shown in equation (5) below: Where σ represents the adjustment factor of the loss function in the semantic segmentation branch, and ω represents the adjustment factor of the regression loss in the object detection branch. This represents the adjustment factor for the classification loss in the object detection branch.

8. A laser point cloud detection and segmentation system, characterized in that, It includes a feature extraction module, a task processing module, and a rate balancing module, among which... The feature extraction module is configured to use a convolutional neural network as a feature extractor to extract semantic features and edge features from the detected image, thereby obtaining semantic features and edge features. The task processing module is configured to, after the convolutional neural network, use two parallel network branches as an object detection branch and a semantic segmentation branch, and process the semantic features and edge features through the object detection branch and the semantic segmentation branch. Specifically, the object detection branch handles object detection tasks, and the semantic segmentation branch handles semantic segmentation tasks. The object detection branch includes a first sub-branch, a second sub-branch, and a third sub-branch. The first sub-branch is used to classify the detected object, the second sub-branch is used to predict the bounding box for calculating depth information, and the third sub-branch is used to estimate the orientation of the detected object. The object detection branch handles the object detection task, including: The first sub-branch determines the category of the detected object using the Softmax function; The second sub-branch uses a regression approach to add a Gaussian kernel to the feature map of the two-dimensional image to obtain a two-dimensional response distribution and a heatmap. The highest point in the two-dimensional response distribution is taken as the object center point, and the target box is obtained based on the distance from the object center point and the predicted center point to the object bounding box. The third sub-branch estimates the orientation of the detected object by calculating the depth information of the feature map of the two-dimensional image and comparing it with the depth information in the target box. The rate balancing module is configured to use a loss function to balance the object detection task and the semantic segmentation task for optimization iteration; wherein the loss function used in the object detection branch includes: The loss function L reg (x,r,g),L cls (x,p) are shown in equations (3) and (4) below: Where M∈{x0,y0,z0,w,h,l,θ} represents the 7 values ​​that need to be regressed for a detected object, namely the center coordinates (x0,y0,z0), length, width and height (w,h,l) and skew angle θ of the target box; Whether the i-th pixel in the feature map of the two-dimensional image corresponds to a ground truth bounding box of a detected object; and These represent the output value of the second sub-branch corresponding to the regression value and the pre-given true labeled value, respectively; and These represent the probability outputs for positive and negative samples, respectively.

Images