Point cloud semantic segmentation network training method, point cloud semantic segmentation method and device

Abstract

The present disclosure provides a point cloud semantic segmentation network training method, a point cloud semantic segmentation method, an apparatus, an electronic device and a storage medium, which can be applied to the technical field of artificial intelligence. The method comprises: mapping multiple groups of point cloud data into initial views respectively to obtain multiple ring-view projection maps; performing partition processing on a first ring-view projection map and a second ring-view projection map respectively based on a preset size to obtain multiple first partition maps and multiple second partition maps; determining multiple first target partition maps from the multiple first partition maps; replacing a second target partition map in the second ring-view projection map with each of the multiple first target partition maps respectively to obtain a mixed projection map; and training an initial network by taking the first ring-view projection map and the mixed projection map as training samples to obtain a point cloud semantic segmentation network.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a point cloud semantic segmentation network training method, a point cloud semantic segmentation method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] With the development of 3D sensing technology, point cloud data is widely used in many fields such as autonomous driving and robot grasping. Deep learning technology, as the mainstream solution for point cloud data analysis, has demonstrated excellent performance in point cloud data processing. Since point cloud data collected by various sensors is usually unlabeled, and the cost of manually labeling data is high, related technologies typically use semi-supervised training methods to build deep neural networks.

[0003] In related technologies, research on semi-supervised training algorithms to improve semantic segmentation tasks mainly focuses on the field of two-dimensional images. However, when this method is directly applied to the segmentation task of three-dimensional point clouds, it will cause the problem of three-dimensional shape distortion, which will indirectly lead to poor semantic segmentation results of point cloud data. Summary of the Invention

[0004] In view of this, the present disclosure provides a point cloud semantic segmentation network training method, a point cloud semantic segmentation method, an apparatus, an electronic device, a readable storage medium, and a computer program product.

[0005] One aspect of this disclosure provides a method for training a point cloud semantic segmentation network, comprising: mapping multiple sets of point cloud data onto an initial view to obtain multiple surround view projection maps; partitioning a first surround view projection map and a second surround view projection map based on a preset size to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps; determining multiple first target partition maps from the multiple first partition maps; replacing a second target partition map in the second surround view projection map with each of the multiple first target partition maps to obtain a hybrid projection map, wherein the second target partition map belongs to multiple second partition maps, and the first target partition map and the second target partition map are in the same position; and using the first surround view projection map and the hybrid projection map as training samples to train an initial network to obtain a point cloud semantic segmentation network.

[0006] According to embodiments of this disclosure, the above-mentioned method of using the first surround-view projection map and the mixed projection map as training samples to train an initial network to obtain a point cloud semantic segmentation network includes: inputting the first surround-view projection map and the mixed projection map into the initial network respectively to obtain a first feature map and a first segmentation result corresponding to the first surround-view projection map, and a second feature map and a second segmentation result corresponding to the mixed projection map; calculating the information entropy loss between the first feature map and the second feature map to obtain a first loss value; calculating the cross-entropy loss between the first segmentation result and the second segmentation result to obtain a second loss value; and using the first loss value and the second loss value to adjust the model parameters of the initial network to finally obtain the point cloud data semantic segmentation network.

[0007] According to embodiments of this disclosure, the calculation of the information entropy loss between the first feature map and the second feature map to obtain a first loss value includes: determining a first sub-feature map related to multiple first target partition maps from the first feature map; splitting the second feature map into a second sub-feature map related to multiple first target partition maps and a third sub-feature map unrelated to multiple first target partition maps; and, if the confidence probability of the first sub-feature map is greater than a preset threshold, using the first sub-feature map and the second sub-feature map as a positive sample pair and the first sub-feature map and the third sub-feature map as a negative sample pair, calculating the information entropy loss between the positive sample pair and the negative sample pair to obtain the first loss value.

[0008] According to embodiments of this disclosure, the calculation of the cross-entropy loss between the first segmentation result and the second segmentation result to obtain a second loss value includes: determining a first sub-segmentation result related to a plurality of first target partition maps from the first segmentation result; determining a second sub-segmentation result related to a plurality of first target partition maps from the second segmentation result; determining a predicted value and a label value based on the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result; and calculating the cross-entropy loss between the predicted value and the label value to obtain the second loss value.

[0009] According to embodiments of this disclosure, determining the predicted value and the label value based on the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result includes: determining the first sub-segmentation result as the label value and the second sub-segmentation result as the predicted value when the confidence probability of the first sub-segmentation result is greater than the confidence probability of the second sub-segmentation result; and determining the first sub-segmentation result as the predicted value and the second sub-segmentation result as the label value when the confidence probability of the first sub-segmentation result is less than the confidence probability of the second sub-segmentation result.

[0010] According to embodiments of this disclosure, the plurality of first target partition maps include a third target partition map, the third target partition map having a real label; wherein, the calculation of the cross-entropy loss between the first segmentation result and the second segmentation result to obtain a second loss value includes: determining a third sub-segmentation result related to the third target partition map and a fourth sub-segmentation result unrelated to the third target partition map but related to the plurality of first target partition maps from the first segmentation result; determining a fifth sub-segmentation result related to the third target partition map and a sixth sub-segmentation result unrelated to the third target partition map but related to the plurality of first target partition maps from the second segmentation result; calculating the cross-entropy loss between the third sub-segmentation result and the real label to obtain a third loss value; calculating the cross-entropy loss between the fourth sub-segmentation result and the sixth sub-segmentation result to obtain a fourth loss value; and determining the second loss value based on the third loss value and the fourth loss value.

[0011] According to embodiments of this disclosure, the initial network includes an encoder and a decoder; wherein, the step of inputting the first surround-view projection map and the mixed projection map into the initial network to obtain a first feature map and a first segmentation result corresponding to the first surround-view projection map, and a second feature map and a second segmentation result corresponding to the mixed projection map, includes: inputting the first surround-view projection map and the mixed projection map into the encoder to obtain a first image feature corresponding to the first surround-view projection map and a second image feature corresponding to the mixed projection map; and inputting the first image feature and the second image feature into the decoder to obtain the first feature map and the first segmentation result corresponding to the first surround-view projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map.

[0012] According to embodiments of this disclosure, the above-described mapping of multiple sets of point cloud data onto an initial view to obtain multiple surround-view projection maps includes: for each set of point cloud data, performing polar coordinate transformation on the three-dimensional coordinate data of each point in the point cloud data to obtain polar coordinate data of each point in the point cloud data; based on the polar coordinate data of each point in the point cloud data, mapping the multiple points in the point cloud data onto multiple grids in the initial view; for each grid in the initial view, determining the feature data of the grid based on the three-dimensional coordinate data and polar coordinate data of the points in the grid; and constructing the surround-view projection map based on the feature data of the multiple grids.

[0013] Another aspect of this disclosure provides a point cloud semantic segmentation method, comprising: mapping target point cloud data onto an initial view to obtain a surround view projection map; and inputting the surround view projection map into a point cloud semantic segmentation network to obtain a semantic segmentation feature map of the target point cloud data; wherein the point cloud semantic segmentation network is trained using the point cloud semantic segmentation network training method described above.

[0014] Another aspect of this disclosure provides a point cloud semantic segmentation network training apparatus, comprising: a first mapping module for mapping multiple sets of point cloud data onto an initial view to obtain multiple surround view projection maps; a first processing module for partitioning the first surround view projection map and the second surround view projection map based on a preset size to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps; a determination module for determining multiple first target partition maps from the multiple first partition maps; a second processing module for replacing the second target partition map in the second surround view projection map with each of the first target partition maps in the multiple first target partition maps to obtain a hybrid projection map, wherein the second target partition map belongs to multiple second partition maps, and the positions of the first target partition map and the second target partition map are the same; and a training module for using the first surround view projection map and the hybrid projection map as training samples to train the initial network to obtain a point cloud semantic segmentation network.

[0015] Another aspect of this disclosure provides a point cloud semantic segmentation apparatus, comprising: a second mapping module for mapping target point cloud data onto an initial view to obtain a surround view projection map; and a third processing module for inputting the surround view projection map into a point cloud semantic segmentation network to obtain a semantic segmentation feature map of the target point cloud data; wherein the point cloud semantic segmentation network is trained using the point cloud semantic segmentation network training method described above.

[0016] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0017] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0018] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0019] According to embodiments of this disclosure, when training a point cloud semantic segmentation network, point cloud data can be mapped to a surround-view projection map, and the first and second surround-view projection maps can be partitioned and mixed. Specifically, a portion of the first surround-view projection map is used to replace the corresponding portion of the second surround-view projection map, resulting in a mixed projection map. Then, the mixed projection map and the first surround-view projection map can be used to train an initial network to ultimately obtain the point cloud semantic segmentation network. By using partitioning and mixing, the partitioned portion can be forcibly decoupled from the background, effectively improving data richness, reducing the network's dependence on background and global information when predicting local regions, and improving the network's recognition ability. Simultaneously, partitioning and mixing can effectively preserve the 3D shape of the original point cloud projection on the surround-view projection map, at least partially overcoming the problems of 3D deformation and shape information loss caused by data augmentation, thus improving the network's robustness. Through the above technical means, the utilization efficiency of hardware resources during network training can be effectively improved. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 The illustration schematically depicts an exemplary system architecture for applying point cloud semantic segmentation network training methods, point cloud semantic segmentation methods, and apparatus according to embodiments of the present disclosure.

[0022] Figure 2 A flowchart illustrating a point cloud semantic segmentation network training method according to an embodiment of the present disclosure is shown.

[0023] Figure 3 The diagram illustrates a training process for a point cloud semantic segmentation network according to an embodiment of the present disclosure.

[0024] Figure 4A flowchart illustrating a point cloud semantic segmentation method according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A block diagram of a point cloud semantic segmentation network training apparatus according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 6 A block diagram of a point cloud semantic segmentation apparatus according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 7 The diagram illustrates a block diagram of an electronic device suitable for implementing a point cloud semantic segmentation network training method or a point cloud semantic segmentation method according to embodiments of the present disclosure. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0032] In the field of autonomous driving technology, using deep learning to perceive and recognize the surrounding environment is a crucial fundamental research area. However, deep neural networks built using deep learning technology often require a large amount of manually labeled data for training. The cost and time consumption of this manually labeled data are often barriers to improving the performance of deep neural network models. On the other hand, autonomous vehicles can collect a large amount of raw, unlabeled data through various sensors during operation. Therefore, how to effectively utilize this raw, unlabeled data, supplemented by a small amount of labeled data, to train neural networks—that is, using semi-supervised training to improve the recognition and classification performance of neural networks—is an important research task that can increase efficiency and reduce costs in the development of autonomous driving systems.

[0033] In related technologies, research on using semi-supervised training algorithms to improve semantic segmentation tasks mainly focuses on the 2D image domain. Research on semi-supervised training algorithms for 3D point cloud scenes, especially for 3D point cloud semantic segmentation models based on LiDAR scanning results, remains largely unexplored. Due to the modal differences between 2D images and 3D point clouds, semi-supervised training algorithms for point cloud semantic segmentation on 2D images cannot be directly and effectively transferred to 3D point cloud semantic segmentation tasks. For example, when performing semantic segmentation of 3D point clouds using surround view projection maps, conventional 2D image data augmentation methods, such as adding noise, rotation, and scaling, can distort the 3D shape of the 3D point cloud, thus affecting the model's training performance.

[0034] In view of this, the embodiments of this disclosure provide a method for semi-supervised training of a point cloud semantic segmentation network that can effectively utilize a large amount of raw LiDAR point cloud data, supplemented by a small amount of labeled data. In this method, a partitioned hybrid data augmentation strategy is proposed, which increases the recognition difficulty of the model by mixing two different surround view projection maps, reduces the loss of shape information of the 3D point cloud during the data augmentation process, and thus improves the training effect, robustness and reliability of the model.

[0035] Specifically, embodiments of this disclosure provide a point cloud semantic segmentation network training method, a point cloud semantic segmentation method, an apparatus, an electronic device, and a storage medium. The point cloud semantic segmentation network training method includes: mapping multiple sets of point cloud data onto an initial view to obtain multiple surround-view projection maps; partitioning a first surround-view projection map and a second surround-view projection map based on a preset size to obtain multiple first partition maps and multiple second partition maps, wherein the first and second surround-view projection maps belong to multiple surround-view projection maps; determining multiple first target partition maps from the multiple first partition maps; replacing a second target partition map in a second surround-view projection map with each of the multiple first target partition maps to obtain a hybrid projection map, wherein the second target partition map belongs to multiple second partition maps, and the first and second target partition maps are in the same position; and training an initial network using the first surround-view projection maps and the hybrid projection map as training samples to obtain a point cloud semantic segmentation network.

[0036] Figure 1 This illustration schematically depicts an exemplary system architecture applicable to point cloud semantic segmentation network training methods, point cloud semantic segmentation methods, and apparatus according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0037] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105.

[0038] Terminal devices 101, 102, and 103 can be various devices equipped with lidar, or various electronic devices capable of controlling lidar, or various electronic devices capable of storing point cloud data.

[0039] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0040] Server 105 can be a server that provides various services. For example, the server can provide computing and storage resources to support the training process of a point cloud semantic segmentation network.

[0041] It should be noted that the point cloud semantic segmentation network training method or point cloud semantic segmentation method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the point cloud semantic segmentation network training device or point cloud semantic segmentation device provided in this disclosure embodiment can generally be located in server 105. Terminal devices 101, 102, and 103 can collect point cloud data, or terminal devices 101, 102, and 103 can obtain point cloud data collected by other terminal devices through the Internet or other means. This point cloud data can be sent to server 105 through the network so that server 105 can execute the method provided in this disclosure embodiment to train the point cloud semantic segmentation network or perform point cloud semantic segmentation on the point cloud data. The point cloud semantic segmentation network training method or point cloud semantic segmentation method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and can communicate with terminal devices 101, 102, 103 and / or server 105. Accordingly, the point cloud semantic segmentation network training device or point cloud semantic segmentation device provided in this embodiment of the present disclosure can also be located in a server or server cluster that is different from server 105 and can communicate with terminal devices 101, 102, 103 and / or server 105. Alternatively, the point cloud semantic segmentation network training method or point cloud semantic segmentation method provided in this embodiment of the present disclosure can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Accordingly, the point cloud semantic segmentation network training device or point cloud semantic segmentation device provided in this embodiment of the present disclosure can also be located in terminal devices 101, 102, or 103, or in other terminal devices different from terminal devices 101, 102, or 103.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Figure 2 A flowchart illustrating a point cloud semantic segmentation network training method according to an embodiment of the present disclosure is shown.

[0044] like Figure 2As shown, the method includes operations S201 to S205.

[0045] In operation S201, multiple sets of point cloud data are mapped onto the initial view to obtain multiple panoramic projection maps.

[0046] In operation S202, based on a preset size, the first surround view projection map and the second surround view projection map are partitioned to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps.

[0047] In operation S203, multiple first target partition maps are determined from multiple first partition maps.

[0048] In operation S204, each of the multiple first target partition maps is used to replace the second target partition map in the second surround projection map to obtain a mixed projection map. The second target partition map belongs to multiple second partition maps, and the positions of the first target partition map and the second target partition map are the same.

[0049] In operation S205, the first surround projection map and the mixed projection map are used as training samples to train the initial network, resulting in a point cloud semantic segmentation network.

[0050] According to embodiments of this disclosure, point cloud data can be acquired using sensing devices such as rotating scanning lidar. Each set of point cloud data can be configured with a preset Cartesian coordinate system. Each point in the point cloud data can be represented as a three-dimensional coordinate in the Cartesian coordinate system. The center of the Cartesian coordinate system can represent the position of the sensing device when the point cloud data is acquired.

[0051] According to embodiments of this disclosure, point cloud data acquired using a rotating scanning lidar can be distributed within a sphere, and the initial view can be obtained by unfolding the sphere onto an annular surface near a horizontal plane. For each point in the point cloud data, a direction vector for mapping can be determined based on the point's coordinates, and this direction vector can then be used to project the point onto the initial view.

[0052] According to embodiments of this disclosure, partitioning a surround-view projection image based on a preset size can divide the image into multiple rectangular regions. The size of the preset size can be determined based on the dimensions of the surround-view projection image in a specific application scenario, and is not limited thereto. For example, if the resolution of the surround-view projection image is 24×480, it can be partitioned into 16 parts along its length and 6 parts along its width, resulting in a total of 96 partitioned images with a resolution of 4×30.

[0053] According to embodiments of this disclosure, the first and second surround view projection maps can be randomly selected from a plurality of surround view projection maps. The first and second surround view projection maps can have completely different characteristics; that is, the point cloud data corresponding to the first and second surround view projection maps can be collected from different objects in different scenes.

[0054] According to embodiments of this disclosure, the first target partition map can be randomly collected from multiple first partition maps, and the first target partition map can occupy a certain proportion in the first partition map, such as 25%, 30%, etc., which is not limited here.

[0055] According to embodiments of this disclosure, the operation of replacing a second target partition map in a second surround view projection map with each of a plurality of first target partition maps may include: determining a second target partition map from the second surround view projection map based on the position information of the first target partition map, deleting the second target partition map, and then filling the corresponding position with the first target partition map.

[0056] According to embodiments of this disclosure, the method used to train the initial network is not limited, and may include gradient descent, least squares, etc. The training parameters set during initial network training, such as the number of training iterations, batch size, and learning rate, can be set according to specific application scenarios and are not limited here.

[0057] According to embodiments of this disclosure, when training a point cloud semantic segmentation network, point cloud data can be mapped to a surround-view projection map, and the first and second surround-view projection maps can be partitioned and mixed. Specifically, a portion of the first surround-view projection map is used to replace the corresponding portion of the second surround-view projection map, resulting in a mixed projection map. Then, the mixed projection map and the first surround-view projection map can be used to train an initial network to ultimately obtain the point cloud semantic segmentation network. By using partitioning and mixing, the partitioned portion can be forcibly decoupled from the background, effectively improving data richness, reducing the network's dependence on background and global information when predicting local regions, and improving the network's recognition ability. Simultaneously, partitioning and mixing can effectively preserve the 3D shape of the original point cloud projection on the surround-view projection map, at least partially overcoming the problems of 3D deformation and shape information loss caused by data augmentation, thus improving the network's robustness. Through the above technical means, the utilization efficiency of hardware resources during network training can be effectively improved.

[0058] The following is for reference. Figure 3 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0059] According to embodiments of this disclosure, the surround view projection map can be obtained using the method of operation S201. Specifically, operation S201 may include the following operations:

[0060] For each set of point cloud data, polar coordinate transformation is performed on the 3D coordinate data of each point in the point cloud data to obtain the polar coordinate data of each point in the point cloud data; based on the polar coordinate data of each point in the point cloud data, multiple points in the point cloud data are mapped to multiple grids in the initial view; for each grid in the initial view, the feature data of the grid is determined based on the 3D coordinate data and polar coordinate data of the points in the grid; and, based on the feature data of multiple grids, a surround view projection map is constructed.

[0061] According to embodiments of this disclosure, each point in the point cloud data can have three-dimensional coordinate data, namely x, y and z. By performing polar coordinate transformation on the point, the transformed coordinates yaw and pitch in the rotating coordinate system can be obtained, i.e., polar coordinate data.

[0062] According to embodiments of this disclosure, the grid of the initial view can refer to pixel color blocks corresponding to individual pixels in the initial view. For example, if the resolution of the initial view is 20×480, then the initial view can have 9600 pixel color blocks, and correspondingly, the initial view can have 9600 grids.

[0063] According to embodiments of this disclosure, when multiple points are mapped in a raster, the feature data of the point closest to the origin among the multiple points can be taken as the feature data of the raster. The feature data of this point may include three-dimensional coordinate data, polar coordinate data, and data obtained by processing the three-dimensional coordinate data and polar coordinate data, such as reflectance data, depth data, etc.

[0064] Figure 3 The diagram illustrates a training process for a point cloud semantic segmentation network according to an embodiment of the present disclosure.

[0065] like Figure 3 As shown, the training process of a point cloud semantic segmentation network can include a sample preprocessing process and a network iterative training process.

[0066] According to embodiments of this disclosure, during sample preprocessing, a portion of the partitions in the first panoramic projection map can be replaced in the second panoramic projection map to obtain a hybrid projection map. Specific methods can be found in operations S202-S204, and will not be repeated here.

[0067] According to embodiments of this disclosure, the network iterative training process may involve inputting the first all-around projection map and the mixed projection map as sample pairs into the initial network, and adjusting the model parameters of the initial network based on the set loss function and model iterative methods such as gradient descent and least squares, so as to achieve the training of the initial network.

[0068] According to embodiments of this disclosure, the initial network may include an encoder and a decoder.

[0069] According to embodiments of this disclosure, inputting a first surround-view projection map and a hybrid projection map into an initial network to obtain a first feature map and a first segmentation result corresponding to the first surround-view projection map, and a second feature map and a second segmentation result corresponding to the hybrid projection map may include the following operations:

[0070] The first surround view projection map and the mixed projection map are respectively input into the encoder to obtain the first image feature corresponding to the first surround view projection map and the second image feature corresponding to the mixed projection map; and the first image feature and the second image feature are respectively input into the decoder to obtain the first feature map and the first segmentation result corresponding to the first surround view projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map.

[0071] According to embodiments of this disclosure, the encoder can be any feature extraction network, such as ResNet18.

[0072] According to embodiments of this disclosure, the decoder can be any feature upsampling network, such as UpperNet.

[0073] According to embodiments of this disclosure, the network iterative training process may specifically include the following operations:

[0074] The first all-around projection map and the mixed projection map are input into the initial network to obtain the first feature map and the first segmentation result corresponding to the first all-around projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map. The information entropy loss between the first feature map and the second feature map is calculated to obtain the first loss value. The cross-entropy loss between the first segmentation result and the second segmentation result is calculated to obtain the second loss value. The model parameters of the initial network are adjusted using the first loss value and the second loss value to finally obtain the point cloud data semantic segmentation network.

[0075] According to embodiments of this disclosure, the first segmentation result can represent the semantic feature segmentation result of each region in the first panoramic projection map.

[0076] According to embodiments of this disclosure, the first feature map may have the same size as the first surround-view projection map, and regions with different semantic features on the first feature map may have different color features. For example, regions with different semantic features on the first feature map may refer to the areas where people, vehicles, and obstacles are located, respectively, and these three regions may be represented by red, blue, and green, respectively.

[0077] According to embodiments of this disclosure, calculating the information entropy loss between the first feature map and the second feature map to obtain the first loss value may include the following operations:

[0078] From the first feature map, a first sub-feature map related to multiple first target partition maps is determined; the second feature map is split into a second sub-feature map related to multiple first target partition maps and a third sub-feature map unrelated to multiple first target partition maps; and when the confidence probability of the first sub-feature map is greater than a preset threshold, the first sub-feature map and the second sub-feature map are used as positive sample pairs, and the first sub-feature map and the third sub-feature map are used as negative sample pairs, and the information entropy loss between the positive sample pairs and the negative sample pairs is calculated to obtain a first loss value.

[0079] According to embodiments of this disclosure, since the first feature map can have the same size as the first surround view projection map, the first sub-feature map can be determined from the first feature map based on the position information of the first target partition map in the first surround view projection map.

[0080] According to embodiments of this disclosure, the method for calculating the confidence probability of the first sub-feature map is not limited herein; for example, the confidence probability can be determined using the Gaussian formula.

[0081] According to embodiments of this disclosure, the preset threshold can be determined based on specific application scenarios, such as being set to 90%, 95%, etc., without limitation.

[0082] According to embodiments of this disclosure, the method for calculating information entropy loss can be as shown in formula (1):

[0083]

[0084] In equation (1), L1 represents the information entropy loss; fp represents the first sub-feature map; fx represents the second sub-feature map; and fy represents the third sub-feature map.

[0085] According to embodiments of this disclosure, calculating the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value may include the following operations:

[0086] From the first segmentation result, determine the first sub-segmentation result related to multiple first target partition maps; from the second segmentation result, determine the second sub-segmentation result related to multiple first target partition maps; based on the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result, determine the predicted value and the label value; and calculate the cross-entropy loss between the predicted value and the label value to obtain the second loss value.

[0087] According to embodiments of this disclosure, the first segmentation result may have the same size as the first surround view projection map, and thus the first sub-segmentation result can be determined from the first segmentation result based on the position information of the first target partition map in the first surround view projection map.

[0088] According to embodiments of this disclosure, the method for calculating the confidence probabilities of the first sub-segmentation result and the second sub-segmentation result is not limited herein; for example, the confidence probability can be determined using the Gaussian formula.

[0089] According to embodiments of this disclosure, the predicted value and the label value can be determined by comparing the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result. Specifically, if the confidence probability of the first sub-segmentation result is greater than the confidence probability of the second sub-segmentation result, the first sub-segmentation result is determined as the label value and the second sub-segmentation result is determined as the predicted value; if the confidence probability of the first sub-segmentation result is less than the confidence probability of the second sub-segmentation result, the first sub-segmentation result is determined as the predicted value and the second sub-segmentation result is determined as the label value.

[0090] According to embodiments of this disclosure, the cross-entropy loss can be calculated as shown in formula (2):

[0091] L2=∑(ylogy p +(1-y)log(1-y p (2)

[0092] In equation (2), L2 represents the cross-entropy loss; y represents the label value; y p This represents the predicted value.

[0093] According to embodiments of this disclosure, the total loss used for adjusting the model parameters of the initial network can be a weighted sum of information entropy loss and cross-entropy loss, and the weight can be a hyperparameter that can be arbitrarily set by the user during model tuning.

[0094] According to embodiments of this disclosure, a third target partition map may be included in a plurality of first target partition maps, the third target partition map having real labels.

[0095] According to an embodiment of this disclosure, when it is determined that a third target partitioning map exists, calculating the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value may include the following operations:

[0096] From the first segmentation result, determine the third sub-segmentation result related to the third target partition map, and the fourth sub-segmentation result unrelated to the third target partition map but related to multiple first target partition maps; from the second segmentation result, determine the fifth sub-segmentation result related to the third target partition map, and the sixth sub-segmentation result unrelated to the third target partition map but related to multiple first target partition maps; calculate the cross-entropy loss between the third sub-segmentation result and the true label to obtain the third loss value; calculate the cross-entropy loss between the fourth sub-segmentation result and the sixth sub-segmentation result to obtain the fourth loss value; and based on the third loss value and the fourth loss value, determine the second loss value.

[0097] According to the embodiments of this disclosure, by designing the above-described loss function, the network can be trained using the original unlabeled data and a small amount of labeled data, thereby realizing semi-supervised training of the point cloud semantic segmentation network. This reduces the cost of data labeling while ensuring the semantic segmentation effect of the network.

[0098] Figure 4 A flowchart illustrating a point cloud semantic segmentation method according to an embodiment of the present disclosure is shown.

[0099] like Figure 4 As shown, the method includes operations S401 to S402.

[0100] In operation S401, the target point cloud data is mapped onto the initial view to obtain a surround projection map.

[0101] In operation S402, the surround projection map is input into the point cloud semantic segmentation network to obtain the semantic segmentation feature map of the target point cloud data.

[0102] According to embodiments of this disclosure, the point cloud semantic segmentation network can be trained using the methods described in the point cloud semantic segmentation network training method section above, which will not be repeated here.

[0103] Figure 5 A block diagram of a point cloud semantic segmentation network training apparatus according to an embodiment of the present disclosure is shown schematically.

[0104] like Figure 5 As shown, the point cloud semantic segmentation network training device 500 includes a first mapping module 510, a first processing module 520, a determination module 530, a second processing module 540, and a training module 550.

[0105] The first mapping module 510 is used to map multiple sets of point cloud data onto the initial view to obtain multiple panoramic projection maps.

[0106] The first processing module 520 is used to partition the first surround view projection map and the second surround view projection map according to a preset size to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps.

[0107] The determination module 530 is used to determine multiple first target partition maps from multiple first partition maps.

[0108] The second processing module 540 is used to replace the second target partition map in the second surround projection map with each of the multiple first target partition maps to obtain a mixed projection map, wherein the second target partition map belongs to multiple second partition maps and the first target partition map and the second target partition map are in the same position.

[0109] The training module 550 is used to train the initial network using the first all-around projection map and the mixed projection map as training samples to obtain the point cloud semantic segmentation network.

[0110] According to embodiments of this disclosure, when training a point cloud semantic segmentation network, point cloud data can be mapped to a surround-view projection map, and the first and second surround-view projection maps can be partitioned and mixed. Specifically, a portion of the first surround-view projection map is used to replace the corresponding portion of the second surround-view projection map, resulting in a mixed projection map. Then, the mixed projection map and the first surround-view projection map can be used to train an initial network to ultimately obtain the point cloud semantic segmentation network. By using partitioning and mixing, the partitioned portion can be forcibly decoupled from the background, effectively improving data richness, reducing the network's dependence on background and global information when predicting local regions, and improving the network's recognition ability. Simultaneously, partitioning and mixing can effectively preserve the 3D shape of the original point cloud projection on the surround-view projection map, at least partially overcoming the problems of 3D deformation and shape information loss caused by data augmentation, thus improving the network's robustness. Through the above technical means, the utilization efficiency of hardware resources during network training can be effectively improved.

[0111] According to embodiments of this disclosure, training module 550 includes a first training submodule, a second training submodule, a third training submodule, and a fourth training submodule.

[0112] The first training submodule is used to input the first all-around projection map and the mixed projection map into the initial network respectively to obtain the first feature map and the first segmentation result corresponding to the first all-around projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map.

[0113] The second training submodule is used to calculate the information entropy loss between the first feature map and the second feature map to obtain the first loss value.

[0114] The third training submodule is used to calculate the cross-entropy loss between the first and second segmentation results to obtain the second loss value.

[0115] The fourth training submodule is used to adjust the model parameters of the initial network using the first and second loss values ​​to finally obtain the semantic segmentation network for point cloud data.

[0116] According to embodiments of this disclosure, the second training submodule includes a first training unit, a second training unit, and a third training unit.

[0117] The first training unit is used to determine the first sub-feature map associated with multiple first target partition maps from the first feature map.

[0118] The second training unit is used to split the second feature map into a second sub-feature map that is related to multiple first target partition maps and a third sub-feature map that is unrelated to multiple first target partition maps.

[0119] The third training unit is used to calculate the information entropy loss between the positive and negative sample pairs when the confidence probability of the first sub-feature map is greater than a preset threshold, using the first and second sub-feature maps as positive sample pairs and the first and third sub-feature maps as negative sample pairs, to obtain the first loss value.

[0120] According to embodiments of this disclosure, the third training submodule includes a fourth training unit, a fifth training unit, a sixth training unit, and a seventh training unit.

[0121] The fourth training unit is used to determine the first sub-segmentation results associated with multiple first target partition maps from the first segmentation results.

[0122] The fifth training unit is used to determine the second sub-segmentation results associated with multiple first target partition maps from the second segmentation results.

[0123] The sixth training unit is used to determine the predicted value and the label value based on the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result.

[0124] The seventh training unit is used to calculate the cross-entropy loss between the predicted value and the label value, thus obtaining the second loss value.

[0125] According to embodiments of this disclosure, the sixth training unit includes a first training subunit and a second training subunit.

[0126] The first training subunit is used to determine the first sub-segmentation result as the label value and the second sub-segmentation result as the predicted value when the confidence probability of the first sub-segmentation result is greater than the confidence probability of the second sub-segmentation result.

[0127] The second training subunit is used to determine the first sub-segmentation result as the predicted value and the second sub-segmentation result as the label value when the confidence probability of the first sub-segmentation result is less than the confidence probability of the second sub-segmentation result.

[0128] According to embodiments of this disclosure, a third target partition map is included among the plurality of first target partition maps, the third target partition map having real labels.

[0129] According to embodiments of this disclosure, the third training submodule includes an eighth training unit, a ninth training unit, a tenth training unit, an eleventh training unit, and a twelfth training unit.

[0130] The eighth training unit is used to determine, from the first segmentation results, a third sub-segmentation result that is related to the third target partition map and a fourth sub-segmentation result that is unrelated to the third target partition map but is related to multiple first target partition maps.

[0131] The ninth training unit is used to determine, from the second segmentation results, a fifth sub-segmentation result that is related to the third target partition map and a sixth sub-segmentation result that is unrelated to the third target partition map but is related to multiple first target partition maps.

[0132] The tenth training unit is used to calculate the cross-entropy loss between the third sub-segmentation result and the true label, thus obtaining the third loss value.

[0133] The eleventh training unit is used to calculate the cross-entropy loss between the fourth sub-segmentation result and the sixth sub-segmentation result, thus obtaining the fourth loss value.

[0134] The twelfth training unit is used to determine the second loss value based on the third and fourth loss values.

[0135] According to embodiments of this disclosure, the initial network includes an encoder and a decoder.

[0136] According to embodiments of this disclosure, the first training submodule includes a thirteenth training unit and a fourteenth training unit.

[0137] The thirteenth training unit is used to input the first surround projection map and the mixed projection map into the encoder respectively to obtain the first image feature corresponding to the first surround projection map and the second image feature corresponding to the mixed projection map.

[0138] The fourteenth training unit is used to input the first image features and the second image features into the decoder to obtain the first feature map and the first segmentation result corresponding to the first panoramic projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map.

[0139] According to embodiments of this disclosure, the first mapping module 510 includes a first mapping unit, a second mapping unit, a third mapping unit, and a fourth mapping unit.

[0140] The first mapping unit is used to perform polar coordinate transformation on the three-dimensional coordinate data of each point in the point cloud data for each set of point cloud data, so as to obtain the polar coordinate data of each point in the point cloud data.

[0141] The second mapping unit is used to map multiple points in the point cloud data to multiple grids in the initial view based on the polar coordinate data of each point in the point cloud data.

[0142] The third mapping unit is used to determine the feature data of each grid in the initial view based on the three-dimensional coordinate data and polar coordinate data of the points in the grid.

[0143] The fourth mapping unit is used to construct a surround projection map based on the feature data of multiple grids.

[0144] It should be noted that the point cloud semantic segmentation network training device part in the embodiments of this disclosure corresponds to the point cloud semantic segmentation network training method part in the embodiments of this disclosure. For a detailed description of the point cloud semantic segmentation network training device part, please refer to the point cloud semantic segmentation network training method part, which will not be repeated here.

[0145] Figure 6 A block diagram of a point cloud semantic segmentation apparatus according to an embodiment of the present disclosure is shown schematically.

[0146] like Figure 6 As shown, the point cloud semantic segmentation device 600 includes a second mapping module 610 and a third processing module 620.

[0147] The second mapping module 610 is used to map the target point cloud data onto the initial view to obtain a surround projection map.

[0148] The third processing module 620 is used to input the surround projection map into the point cloud semantic segmentation network to obtain the semantic segmentation feature map of the target point cloud data.

[0149] According to embodiments of this disclosure, the point cloud semantic segmentation network can be trained using the methods described in the point cloud semantic segmentation network training method section above, which will not be repeated here.

[0150] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0151] For example, the first mapping module 510, the first processing module 520, the determination module 530, the second processing module 540, and the training module 550, or any multiple of the second mapping module 610 and the third processing module 620, can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first mapping module 510, the first processing module 520, the determining module 530, the second processing module 540, and the training module 550, or at least one of the second mapping module 610 and the third processing module 620, can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first mapping module 510, the first processing module 520, the determining module 530, the second processing module 540, and the training module 550, or at least one of the second mapping module 610 and the third processing module 620, can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0152] Figure 7The diagram illustrates a block diagram of an electronic device suitable for implementing a point cloud semantic segmentation network training method or a point cloud semantic segmentation method according to embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0153] like Figure 7 As shown, a computer electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0154] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0155] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0156] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0157] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0158] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0160] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the point cloud semantic segmentation network training method or point cloud semantic segmentation method provided in the embodiments of this disclosure.

[0161] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0162] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0163] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0165] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for training a point cloud semantic segmentation network, comprising: Multiple sets of point cloud data are mapped onto the initial view to obtain multiple surround view projection maps; Based on a preset size, the first surround view projection map and the second surround view projection map are partitioned to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps; Multiple first target partition maps are determined from multiple first partition maps; By replacing the second target partition map in the second surround-view projection map with each of the multiple first target partition maps, a hybrid projection map is obtained, wherein the second target partition map belongs to multiple second partition maps, and the positions of the first target map and the second target map are the same; and The first surround-view projection map and the hybrid projection map are used as training samples and input into the initial network to train the initial network, thereby obtaining a point cloud semantic segmentation network. The step of using the first surround-view projection map and the mixed projection map as training samples to input into the initial network to train the initial network and obtain a point cloud semantic segmentation network includes: The first surround projection map and the mixed projection map are respectively input into the initial network to obtain a first feature map and a first segmentation result corresponding to the first surround projection map, and a second feature map and a second segmentation result corresponding to the mixed projection map; Calculate the information entropy loss between the first feature map and the second feature map to obtain the first loss value; Calculate the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value; and The model parameters of the initial network are adjusted using the first loss value and the second loss value to finally obtain the semantic segmentation network for the point cloud data.

2. The method according to claim 1, wherein, The step of calculating the information entropy loss between the first feature map and the second feature map to obtain the first loss value includes: From the first feature map, determine the first sub-feature map associated with multiple first target partition maps; The second feature map is split into a second sub-feature map related to multiple first target partition maps and a third sub-feature map unrelated to multiple first target partition maps; and If the confidence probability of the first sub-feature map is greater than a preset threshold, the first sub-feature map and the second sub-feature map are used as positive sample pairs, and the first sub-feature map and the third sub-feature map are used as negative sample pairs. The information entropy loss between the positive sample pairs and the negative sample pairs is calculated to obtain the first loss value.

3. The method according to claim 1, wherein, The calculation of the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value includes: Determine a first sub-segmentation result related to multiple first target partition maps from the first segmentation result; From the second segmentation result, determine the second sub-segmentation result related to multiple first target partition maps; Based on the confidence probabilities of the first and second sub-segmentation results, the predicted and label values ​​are determined; and The cross-entropy loss between the predicted value and the label value is calculated to obtain the second loss value.

4. The method according to claim 3, wherein, The step of determining the predicted value and label value based on the confidence probability of the first sub-segmentation result and the confidence probability of the second sub-segmentation result includes: If the confidence probability of the first sub-segmentation result is greater than the confidence probability of the second sub-segmentation result, then the first sub-segmentation result is determined to be the label value, and the second sub-segmentation result is determined to be the predicted value; and If the confidence probability of the first sub-segmentation result is less than the confidence probability of the second sub-segmentation result, the first sub-segmentation result is determined to be the predicted value, and the second sub-segmentation result is determined to be the label value.

5. The method according to claim 1, wherein, The first target partition map includes a third target partition map, which has a real label; The step of calculating the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value includes: From the first segmentation result, a third sub-segmentation result related to the third target partition map and a fourth sub-segmentation result unrelated to the third target partition map but related to multiple first target partition maps are determined; From the second segmentation result, a fifth sub-segmentation result related to the third target partition map and a sixth sub-segmentation result unrelated to the third target partition map but related to multiple first target partition maps are determined; Calculate the cross-entropy loss between the third sub-segmentation result and the true label to obtain the third loss value; Calculate the cross-entropy loss between the fourth sub-segmentation result and the sixth sub-segmentation result to obtain the fourth loss value; and The second loss value is determined based on the third loss value and the fourth loss value.

6. The method according to claim 1, wherein, The initial network includes an encoder and a decoder; The step of inputting the first surround-view projection map and the hybrid projection map into the initial network to obtain a first feature map and a first segmentation result corresponding to the first surround-view projection map, and a second feature map and a second segmentation result corresponding to the hybrid projection map, includes: The first surround view projection map and the hybrid projection map are input into the encoder respectively to obtain a first image feature corresponding to the first surround view projection map and a second image feature corresponding to the hybrid projection map; and The first image feature and the second image feature are respectively input into the decoder to obtain the first feature map and the first segmentation result corresponding to the first surround projection map, and the second feature map and the second segmentation result corresponding to the mixed projection map.

7. The method according to claim 1, wherein, The process of mapping multiple sets of point cloud data onto the initial view to obtain multiple surround view projection maps includes: For each set of point cloud data, polar coordinate transformation is performed on the three-dimensional coordinate data of each point in the point cloud data to obtain the polar coordinate data of each point in the point cloud data; Based on the polar coordinate data of each point in the point cloud data, multiple points in the point cloud data are mapped to multiple grids in the initial view respectively; For each grid cell in the initial view, feature data of the grid cell is determined based on the three-dimensional coordinate data and polar coordinate data of the points in the grid cell; and The surround projection map is constructed based on the feature data of multiple grids.

8. A point cloud semantic segmentation method, comprising: The target point cloud data is mapped onto the initial view to obtain a surround view projection map; as well as The surround projection map is input into the point cloud semantic segmentation network to obtain the semantic segmentation feature map of the target point cloud data; The point cloud semantic segmentation network is trained using the point cloud semantic segmentation network training method according to any one of claims 1 to 7.

9. A point cloud semantic segmentation network training device, comprising: The first mapping module is used to map multiple sets of point cloud data onto the initial view to obtain multiple surround view projection maps; The first processing module is used to partition the first surround view projection map and the second surround view projection map according to a preset size to obtain multiple first partition maps and multiple second partition maps, wherein the first surround view projection map and the second surround view projection map belong to multiple surround view projection maps. A determining module is configured to determine multiple first target partition maps from multiple first partition maps; The second processing module is configured to replace the second target partition map in the second surround-view projection map with each of the multiple first target partition maps to obtain a hybrid projection map, wherein the second target partition map belongs to multiple second partition maps, and the positions of the first target map and the second target map are the same; and The training module is used to input the first all-around projection map and the mixed projection map as training samples into the initial network to train the initial network and obtain a point cloud semantic segmentation network, including: The first surround projection map and the mixed projection map are respectively input into the initial network to obtain a first feature map and a first segmentation result corresponding to the first surround projection map, and a second feature map and a second segmentation result corresponding to the mixed projection map; Calculate the information entropy loss between the first feature map and the second feature map to obtain the first loss value; Calculate the cross-entropy loss between the first segmentation result and the second segmentation result to obtain the second loss value; and The model parameters of the initial network are adjusted using the first loss value and the second loss value to finally obtain the semantic segmentation network for the point cloud data.

10. A point cloud semantic segmentation device, comprising: The second mapping module is used to map the target point cloud data onto the initial view to obtain a surround view projection map; as well as The third processing module is used to input the surround projection map into the point cloud semantic segmentation network to obtain the semantic segmentation feature map of the target point cloud data; The point cloud semantic segmentation network is trained using the point cloud semantic segmentation network training method according to any one of claims 1 to 7.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 8.

12. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

13. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 8.

Images