Point cloud semantic segmentation method, computer equipment, storage medium and intelligent device

By rastering and feature extraction of three-dimensional point clouds and combining with semantic segmentation networks, accurate semantic segmentation of common category goals is achieved, which solves the problem that traditional methods cannot segment common category goals and improves the safety of autonomous driving.

CN117392393BActive Publication Date: 2024-11-26安徽蔚来智驾科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311707960.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2024-11-26
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

Traditional target detection, tracking and prediction methods cannot semantically segment the targets of the general category, affecting the driving safety of autonomous vehicles.

Method used

Using point cloud semantic segmentation method, multiple three-dimensional point cloud rastering are obtained by rasterizing the three-dimensional point cloud, downsampling and feature extraction are carried out at multiple different scales, fusing to form two-dimensional raster features, and using pre-pre- The semantic segmentation network is set for semantic segmentation to obtain the semantic categories of each point cloud raster.

Benefits of technology

The feature expression ability of two-dimensional grid features is significantly improved, the accurate semantic segmentation of general category targets is achieved, the defect of being unable to segment non-specific category targets in the prior art, and the safety of autonomous driving is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117392393B_ABST
    Figure CN117392393B_ABST
Patent Text Reader

Abstract

The present application relates to the field of autonomous driving technology, and specifically provides a point cloud semantic segmentation method, computer equipment, storage medium and intelligent device, aiming to solve the problem of accurately semantically segmenting targets of general categories. To this end, the method provided by the present application includes rasterizing a three-dimensional point cloud to form a plurality of three-dimensional point cloud grids; for each point cloud grid, downsampling the three-dimensional grid features of the point cloud grid at a plurality of different scales to obtain three-dimensional sampling features of each scale, and converting the three-dimensional sampling features of each scale into two-dimensional sampling features and fusing all two-dimensional sampling features to form two-dimensional grid features; semantic segmentation is performed according to the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid. Based on the above method, the semantic category of each point grid can be accurately obtained, so that the semantic segmentation of the general category target can be accurately completed based on the semantic category of each point grid.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a point cloud semantic segmentation method, computer equipment, storage medium and intelligent device. Background Art

[0002] When controlling a vehicle for autonomous driving, devices such as LiDAR are usually used to collect point cloud data in the vehicle's surrounding environment. The semantics, position, and motion information of obstacles in the environment are perceived based on the point cloud data, and the vehicle is controlled based on the perception results to ensure vehicle safety. Traditional target detection, tracking, and prediction methods can only perform semantic segmentation on specific categories of targets such as motor vehicles and pedestrians, but cannot perform semantic segmentation on general categories (various categories in all scenarios), which affects the driving safety of autonomous vehicles.

[0003] Accordingly, the art needs a new technical solution to solve the above problems. Summary of the invention

[0004] In order to overcome the above-mentioned defects, the present application is proposed to provide a point cloud semantic segmentation method, a computer device, a storage medium and an intelligent device that solve or at least partially solve the technical problem of how to accurately perform semantic segmentation on targets of general categories.

[0005] In a first aspect, a point cloud semantic segmentation method is provided, the method comprising:

[0006] Performing rasterization processing on the three-dimensional point cloud to form a plurality of three-dimensional point cloud grids;

[0007] For each point cloud grid, down-sampling the three-dimensional grid features of the point cloud grid at multiple different scales is performed to obtain three-dimensional sampling features at each scale, and

[0008] For each point cloud grid, converting the three-dimensional sampling features of each scale of the point cloud grid into two-dimensional sampling features of each scale respectively, and fusing all the two-dimensional sampling features of the point cloud grid to form a two-dimensional grid feature of the point cloud grid;

[0009] Semantic segmentation is performed according to the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid.

[0010] In a technical solution of the above-mentioned point cloud semantic segmentation method, the fusing of all two-dimensional sampling features of the point cloud grid includes:

[0011] According to the order of the scales from high to low, the two-dimensional sampling features of each scale are fused in turn, so that the scale of the two-dimensional grid feature formed by the fusion is the same as the scale of the two-dimensional sampling feature of the smallest scale.

[0012] In a technical solution of the above-mentioned point cloud semantic segmentation method, before fusing all two-dimensional sampling features of the point cloud grid, the method further includes:

[0013] Obtaining a two-dimensional sampling feature of a preset scale as a target feature;

[0014] A feature pyramid network is used to extract the target features to obtain new features;

[0015] The new feature is used as the final two-dimensional sampling feature of the preset scale.

[0016] In a technical solution of the above-mentioned point cloud semantic segmentation method, the feature pyramid network is used to extract the target features, including:

[0017] A feature pyramid network is used to downsample the target features at multiple scales to obtain downsampled features at each scale, and

[0018] According to the order of the scales from high to low, the downsampled features of each scale are fused in turn, and the fusion result is used as the new feature;

[0019] The new feature has the same scale as the target feature.

[0020] In a technical solution of the above-mentioned point cloud semantic segmentation method, the semantic segmentation is performed according to the two-dimensional grid features of each point cloud grid, including:

[0021] Use the preset semantic segmentation network and perform semantic segmentation based on the two-dimensional grid features of each point cloud grid;

[0022] The preset semantic segmentation network is trained in the following way:

[0023] Performing rasterization processing on the three-dimensional point cloud samples to form a plurality of three-dimensional point cloud grids;

[0024] For each point cloud grid, respectively obtain the semantic category of each point cloud in the point cloud grid, and set the semantic category label of the point cloud grid according to the first priority of each semantic category in all semantic categories of the point cloud grid;

[0025] The preset semantic segmentation network is trained according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

[0026] In a technical solution of the above-mentioned point cloud semantic segmentation method, setting the semantic category label of the point cloud grid according to the first priority of each semantic category among all semantic categories of the point cloud grid includes:

[0027] From all semantic categories of the point cloud grid, a semantic category with a first priority that satisfies a first preset condition is selected as a target semantic category;

[0028] According to the target semantic category, a semantic category label of the point cloud grid is set.

[0029] In a technical solution of the above-mentioned point cloud semantic segmentation method, the step of selecting a semantic category with a first priority that satisfies a first preset condition as the target semantic category includes:

[0030] If the first priorities of multiple semantic categories in all semantic categories of the point cloud grid all meet the first preset condition,

[0031] Then, from the multiple semantic categories whose first priorities all satisfy the first preset condition, a semantic category whose point cloud quantity satisfies the second preset condition is selected as the target semantic category.

[0032] In a technical solution of the above-mentioned point cloud semantic segmentation method, after selecting a semantic category whose number of point clouds meets the second preset condition as the target semantic category, the method further includes setting the semantic category label of the point cloud grid in the following manner:

[0033] Determine whether the number ratio of point clouds belonging to the target semantic category in the point cloud grid among all point clouds in the point cloud grid is greater than a preset ratio threshold;

[0034] If yes, setting the semantic category label according to the target semantic category;

[0035] If not, the semantic category label is set according to the ratio of the number of point clouds belonging to each semantic category in the point cloud grid to all point clouds in the point cloud grid.

[0036] In a technical solution of the above-mentioned point cloud semantic segmentation method, the training of the preset semantic segmentation network includes:

[0037] According to the second priority of each semantic category, a weight corresponding to each semantic category is set respectively;

[0038] According to the weight corresponding to each semantic category, the loss function corresponding to each semantic category is weighted and calculated to form the target loss function;

[0039] The target loss function is adopted, and the preset semantic segmentation network is trained according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

[0040] In a technical solution of the above-mentioned point cloud semantic segmentation method, the loss function corresponding to each semantic category is obtained by the following method:

[0041] For each semantic category, the cross entropy loss function and the Lovasz loss function are used to respectively construct the first loss function and the second loss function corresponding to the semantic category, and

[0042] For each semantic category, a weighted sum calculation is performed on the first loss function and the second loss function of the semantic category to obtain a final loss function of the semantic category.

[0043] In a second aspect, a computer device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the method described in any one of the technical solutions of the above-mentioned point cloud semantic segmentation method.

[0044] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned point cloud semantic segmentation method.

[0045] In a fourth aspect, a smart device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned point cloud semantic segmentation method is implemented.

[0046] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0047] In implementing the technical solution of the point cloud semantic segmentation method provided in the present application, the three-dimensional point cloud can be rasterized to form multiple three-dimensional point cloud grids. For each point cloud grid, the three-dimensional grid features of the point cloud grid are downsampled at multiple different scales to obtain three-dimensional sampling features of each scale. For each point cloud grid, the three-dimensional sampling features of each scale of the point cloud grid are converted into two-dimensional sampling features of each scale, and all two-dimensional sampling features of the point cloud grid are fused to form two-dimensional grid features of the point cloud grid. Finally, semantic segmentation is performed based on the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid. Based on the above implementation scheme, the feature expression ability of the two-dimensional grid features can be significantly improved, thereby improving the accuracy of semantic segmentation based on the two-dimensional grid features of the point cloud grid. At the same time, the above implementation scheme can obtain the semantic category of each point cloud grid and realize grid-level semantic segmentation. In this way, when performing semantic segmentation of the target, even if the target is not a specific category such as a motor vehicle or a pedestrian, the semantic category of the target can be accurately obtained based on the semantic category of each point cloud grid occupied by the target, thereby overcoming the defect in the prior art that semantic segmentation of targets of general categories cannot be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0049] Figure 1 is a schematic flow chart of the main steps of a point cloud semantic segmentation method according to an embodiment of the present application;

[0050] Figure 2 It is a schematic flow chart of main steps of a method for preprocessing two-dimensional sampling features of a preset scale before fusing the two-dimensional sampling features according to an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of fusing two-dimensional sampling features according to an embodiment of the present application;

[0052] Figure 4 It is a flowchart of the main steps of a method for training a preset semantic segmentation network according to an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of training a preset semantic segmentation network and performing semantic segmentation using the preset semantic segmentation network according to an embodiment of the present application;

[0054] Figure 6is a flowchart of main steps of a method for training a preset semantic segmentation network according to another embodiment of the present application;

[0055] Figure 7 It is a main structural diagram of a computer device according to an embodiment of the present application.

[0056] Figure 8 It is a schematic diagram of the main structure of a smart device according to an embodiment of the present application.

[0057] List of reference numerals:

[0058] 11: storage device; 12: processor; 21: memory; 22: processor. DETAILED DESCRIPTION

[0059] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0060] In the description of this application, "processor" may include hardware, software or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc.

[0061] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0062] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0063] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0064] The following is a description of an embodiment of the point cloud semantic segmentation method provided in this application.

[0065] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a point cloud semantic segmentation method according to an embodiment of the present application. Figure 1 As shown, the point cloud semantic segmentation method in the embodiment of the present application mainly includes the following steps S101 to S104.

[0066] Step S101: performing rasterization processing on the three-dimensional point cloud to form a plurality of three-dimensional point cloud grids.

[0067] The three-dimensional point cloud may be a point cloud collected by an environment sensing device, which is a device capable of collecting three-dimensional point clouds, including but not limited to a laser radar, a millimeter wave radar, etc. The three-dimensional coordinate system of the three-dimensional point cloud may be a device coordinate system of the environment sensing device, such as a laser radar coordinate system.

[0068] In this embodiment, a conventional rasterization processing method in the field of point cloud processing technology can be used to perform rasterization processing on the three-dimensional point cloud, and the three-dimensional point cloud can be divided into multiple point cloud grids, and each point cloud grid is still three-dimensional.

[0069] Step S102: for each point cloud grid, down-sampling the three-dimensional grid features of the point cloud grid at multiple different scales is performed to obtain three-dimensional sampling features at each scale.

[0070] The three-dimensional grid feature refers to the feature information of the point cloud grid, which can reflect or characterize the features of all three-dimensional point clouds in the point cloud grid as a whole. In some embodiments, the feature mean of all three-dimensional point clouds in the point cloud grid can be obtained, and the feature mean is used as the grid feature of the point cloud grid. Based on this embodiment, the grid feature that can reflect or characterize the features of all three-dimensional point clouds in the point cloud grid as a whole can be obtained conveniently and accurately.

[0071] In this embodiment, a conventional downsampling method in the field of point cloud processing technology can be used to downsample the three-dimensional grid features of the point cloud grid at different scales. In addition, those skilled in the art can flexibly set the scale of downsampling according to actual needs, and this embodiment does not specifically limit it. For example, the three-dimensional grid features can be downsampled at 8 times, 4 times, and 2 times the scale.

[0072] Step S103: for each point cloud grid, convert the three-dimensional sampling features of each scale of the point cloud grid into two-dimensional sampling features of each scale respectively, and fuse all the two-dimensional sampling features of the point cloud grid to form a two-dimensional grid feature of the point cloud grid.

[0073] In this embodiment, the feature information of the height dimension in the three-dimensional sampling feature can be merged with the information of the depth (channel) dimension to form a two-dimensional sampling feature. It can also be understood that the three-dimensional sampling feature is converted into a two-dimensional sampling feature through high compression. In some embodiments, a preset two-dimensional grid feature extraction network can be used to perform the above-mentioned high compression on the three-dimensional sampling feature to obtain a two-dimensional sampling feature. This embodiment does not specifically limit the network structure and construction method of the two-dimensional grid feature extraction network, as long as it can convert the three-dimensional sampling feature into a two-dimensional sampling feature through high compression.

[0074] When fusing all the two-dimensional sampling features of the point cloud grid, since the scales of the two-dimensional sampling features are different, each two-dimensional sampling feature can be converted to the same scale and then fused to form a new two-dimensional sampling feature. Among them, the fusion method includes but is not limited to feature splicing, feature addition, etc.

[0075] Step S104: performing semantic segmentation according to the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid respectively.

[0076] Since the two-dimensional grid feature is obtained by integrating two-dimensional sampling features of multiple scales, its feature information actually contains feature information of multiple levels and has a high feature expression capability. Therefore, semantic segmentation based on the two-dimensional grid feature can significantly improve the accuracy of semantic segmentation, thereby accurately obtaining the semantic category of each point cloud grid.

[0077] Based on the methods described in steps S101 to S104 above, the semantic category of each point cloud grid can be accurately obtained. Based on this, when semantically segmenting the target, regardless of whether the target is a specific category such as a motor vehicle or a pedestrian, the semantic category of the target can be accurately obtained based on the semantic category of each point cloud grid occupied by the target.

[0078] In an application scenario according to an embodiment of the present application, a laser radar is installed on the vehicle, and the laser radar can collect three-dimensional point clouds in the surrounding environment. When it is necessary to control the vehicle to perform automatic driving, the semantic segmentation method according to the embodiment of the present application can be used to perform semantic segmentation on the three-dimensional point cloud collected by the laser radar to obtain the semantic category of each point cloud grid, and then the semantic category of each target in the environment is obtained according to the semantic category of the point cloud grid. Among them, the semantic categories may include specific categories such as motor vehicles and pedestrians, and may also include non-specific categories such as cones, railings, and cartons. After obtaining the semantic category of each target in the environment, identify which are obstacles and which are not obstacles, plan the vehicle's driving path according to information such as the location of the obstacles, and control the vehicle to automatically drive according to the driving path.

[0079] The above steps S103 and S104 are further explained below.

[0080] 1. Explain the above step S103.

[0081] In some implementations of the above step S103, the two-dimensional sampling features of each scale may be fused in order from high to low scales, so that the scale of the fused two-dimensional grid feature is the same as that of the two-dimensional sampling feature of the smallest scale.

[0082] In this embodiment, the scale can be divided into a maximum scale, a minimum scale and an intermediate scale according to the size of the scale, and the intermediate scale is a scale between the maximum and minimum scales. If the number of scales is 2, it only needs to be divided into the maximum and minimum scales, and the intermediate scale is not included. The fusion method of the two-dimensional sampling features of the maximum, intermediate and minimum scales is described below.

[0083] 1. Maximum scale two-dimensional sampling features

[0084] In this embodiment, the next level scale of the maximum scale can be obtained, and the "two-dimensional sampling features of the maximum scale" are upsampled according to the next level scale, so that the upsampled features have the same scale as the two-dimensional sampling features of the next level scale. The upsampled features are then fused with the two-dimensional sampling features of the next level scale, and the fusion result is used as a new two-dimensional sampling feature of the next level scale.

[0085] 2. Two-dimensional sampling features at intermediate scales

[0086] In this embodiment, the two-dimensional sampling features of each intermediate scale are fused in descending order of scale.

[0087] For the first intermediate scale (that is, the largest intermediate scale), according to the description in point 1 above, it will be fused with the upsampled features of the "two-dimensional sampling features of the largest scale" to form a new two-dimensional sampling feature. After that, the next level scale of the first intermediate scale can be obtained (it may still be an intermediate scale, or it may be the smallest scale), and the "two-dimensional sampling features of the first intermediate scale" can be upsampled according to this next level scale, so that the upsampled features have the same scale as the two-dimensional sampling features of the next level scale. Then, the upsampled features are fused with the two-dimensional sampling features of the next level scale, and the fusion result is used as the new two-dimensional sampling features of the next level scale.

[0088] For non-first intermediate scales, the fusion process is similar to that of the first intermediate scale mentioned above. The new two-dimensional sampling features are first upsampled according to the next level scale, and then fused with the two-dimensional sampling features of the next level scale. It will not be repeated here.

[0089] 3. Minimum scale two-dimensional sampling features

[0090] In this embodiment, the two-dimensional sampling features of the previous scale (which may be an intermediate scale or a maximum scale) can be upsampled according to the minimum scale (if the previous scale is an intermediate scale, the two-dimensional sampling features are new two-dimensional sampling features obtained by the method described in the above point 2), so that the upsampled features have the same scale as the two-dimensional sampling features of the minimum scale, and the upsampled features are fused with the two-dimensional sampling features of the minimum scale, and the fusion result is the desired two-dimensional grid feature. It should be noted that the next scale of the last intermediate scale is the minimum scale. Therefore, after completing the fusion of the last intermediate scale according to the above point 2, there is no need to fuse the two-dimensional sampling features of the minimum scale again according to the above point 3. If the intermediate scale is not included, after completing the fusion of the maximum scale according to the above point 1, there is no need to fuse the two-dimensional sampling features of the minimum scale again according to the above point 3.

[0091] At this point, the fusion of the two-dimensional sampling features of each scale is completed, and the scale of the fused two-dimensional grid features is made the same as the scale of the two-dimensional sampling features of the smallest scale.

[0092] In some implementations of the above step S103, before fusing all the two-dimensional sampling features of the point cloud grid, Figure 2 The following steps S201 to S203 are shown to pre-process the two-dimensional sampling features of a preset scale to add more layers of features.

[0093] Step S201: Acquire a two-dimensional sampling feature of a preset scale as a target feature.

[0094] Those skilled in the art can flexibly select one or more scales from all scales as preset scales according to actual needs. In some preferred embodiments, the maximum scale can be selected as the preset scale. Based on this, when the two-dimensional sampling features of each scale are fused in order from high to low in the above step S103, the multi-level features added by preprocessing the two-dimensional sampling features of the maximum scale can be fused with the two-dimensional sampling features of each other scale in turn, so as to improve the feature richness of the two-dimensional grid features finally formed by fusion.

[0095] Step S202: Using a Feature Pyramid Network (FPN) to extract target features to obtain new features.

[0096] In this embodiment, a feature pyramid network with a conventional structure can be used. This embodiment does not specifically limit its structure, as long as it can extract target features and obtain new features rich in multi-level features.

[0097] Step S203: taking the new feature as the final two-dimensional sampling feature of the preset scale.

[0098] Based on the method described in steps S201 to S203 above, more levels of features can be added to the two-dimensional sampling features of a preset scale to enhance the feature richness of the two-dimensional grid features finally fused, thereby significantly improving the feature expression capability of the two-dimensional grid features.

[0099] The following is combined with Figure 3 The fusion process of two-dimensional sampling features is briefly explained.

[0100] like Figure 3 As shown in the figure, the sparse S8 feature, sparse S4 feature, and sparse S2 feature represent the three-dimensional sampling features obtained after downsampling the three-dimensional grid features of the point cloud grid by 8 times, 4 times, and 2 times, respectively. The sparse S8 feature, sparse S4 feature, and sparse S2 feature are converted into two-dimensional sampling features E8, E4, and E2 ( Figure 3 The high compression method is the same as the high compression method in the aforementioned step S103, and will not be described in detail here.

[0101] For the 8-fold downsampled 2D sampling feature E8, it is used as the preset 2D sampling feature, and the feature pyramid network is used to extract the new feature, namely, the D8 feature. The next level scale of the 2D sampling feature E8 is 4 times the scale, and the D8 feature is upsampled according to the 4 times scale, and the upsampled feature is fused with the 2D sampling feature E4, and the fusion result is used as the new 2D sampling feature of the 4 times scale, namely, the D4 feature.

[0102] For the D4 feature with a downsampling scale of 4 times, its next level scale is 2 times. The D4 feature is upsampled according to the 2 times scale, and the upsampled feature is fused with the two-dimensional sampling feature E2. The fusion result is used as the new two-dimensional sampling feature of this 2 times scale, that is, the D2 feature. The D2 feature is also the two-dimensional grid feature to be obtained.

[0103] The above step S202 is further explained below.

[0104] In some implementations of the above step S202, a feature pyramid network can be used to downsample the target feature at multiple scales to obtain downsampled features of each scale, and then the downsampled features of each scale are fused in order from high to low scale, and the fusion result is used as a new feature, and the new feature has the same scale as the target feature.

[0105] The downsampled features of each scale obtained by using the feature pyramid network for the target feature are the features of each level, so by obtaining the downsampled features of multiple scales, the features of multiple levels can be obtained. Among them, the downsampled features are still two-dimensional, and the method of fusing the downsampled features of each scale in order from high to low scales is the same as the method of fusing the two-dimensional sampling features of each scale in order from high to low scales in the aforementioned step S103, and will not be repeated here.

[0106] 2. Explain the above step S104.

[0107] In some implementations of the above step S104, a preset semantic segmentation network is used, and semantic segmentation is performed according to the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid. In this implementation, a network with a conventional structure in the field of artificial intelligence technology (such as a convolutional neural network, etc.) can be used to construct a semantic segmentation network, and this implementation does not specifically limit its network structure.

[0108] In this embodiment, Figure 4 The following steps S301 to S304 are shown to train a preset semantic segmentation network so that the trained semantic segmentation network can be used to perform semantic segmentation according to the two-dimensional grid features of each point cloud grid.

[0109] Step S301: rasterize the three-dimensional point cloud samples to form a plurality of three-dimensional point cloud grids. In this embodiment, the conventional rasterization processing method in the field of point cloud processing technology can also be used to rasterize the three-dimensional point cloud samples, and the three-dimensional point cloud samples are divided into a plurality of point cloud grids, each of which is still three-dimensional.

[0110] Step S302: for each point cloud grid, respectively obtain the semantic category of each point cloud in the point cloud grid, that is, obtain the semantic category of each point cloud in the point cloud grid one by one.

[0111] Step S303: setting a semantic category label of the point cloud grid according to the first priority of each semantic category among all semantic categories of the point cloud grid.

[0112] In this embodiment, the importance of each semantic category target to the safe driving of the vehicle can be obtained, and the first priority of different semantic categories can be set according to the importance. The higher the importance, the greater the first priority, and vice versa.

[0113] For example, five first priorities L0 to L4 can be divided from high to low. The semantic categories of L0 may include pedestrians, pedestrian accessories, cyclists, two-wheeled vehicles, animals, etc. The semantic categories of L1 may include vehicles, etc. The semantic categories of L2 may include cones, short columns, long columns, tree stumps, fences, vegetation, curbs, signs, other permanent obstacles and non-permanent obstacles, etc. The semantic categories of L3 may include potholes, speed bumps, etc. The semantic categories of L4 may include drivable roads, walkable roads, filterable targets (rain, snow, fog, dust, etc.), etc.

[0114] In this step, the importance of each point cloud in the point cloud grid to the safe driving of the vehicle can be distinguished according to the first priority of the semantic category, and then the first priorities of all semantic categories of the point cloud grid can be compared, and a semantic category with the largest or larger first priority can be selected as the semantic category label of the point cloud grid.

[0115] Step S304: training a preset semantic segmentation network according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

[0116] In this embodiment, a conventional network training method in the field of machine learning technology can be used to train a preset semantic segmentation network based on the three-dimensional point cloud samples and the semantic category labels of each point cloud grid. For example, the three-dimensional point cloud samples and the semantic category labels of each point cloud grid are input into the semantic segmentation network, the loss value of the network is calculated by forward propagation, the parameter gradient of the network parameters is calculated according to the loss value, and the network parameters are updated according to the parameter gradient back propagation until the semantic segmentation network meets the convergence condition and stops training.

[0117] Based on the method described in steps S301 to S304 above, the semantic category labels of the point cloud grid can be set according to the importance of the targets of each semantic category to the safe driving of the vehicle (represented by the first priority). Compared with setting the semantic category labels according to the number of point clouds of each semantic category, this method can prevent the trained semantic segmentation network from missing targets with higher importance.

[0118] For example, when a point cloud grid contains point clouds with semantic categories of pedestrians and ground at the same time, and the number of ground point clouds is relatively large, if the semantic category label of the point cloud grid is set according to the number of point clouds, the semantic category label will be set to the ground. After the semantic segmentation network is trained using the point cloud grid and its semantic category label, when faced with a point cloud grid that also contains point clouds with semantic categories of pedestrians and ground and the number of ground point clouds is relatively large (for example, a pedestrian only shows one toe in this point cloud grid), this point cloud grid will be predicted as the ground, not a pedestrian. However, pedestrians are very important to the safe driving of vehicles. If they are not accurately detected, it is very likely that the vehicle will collide with the pedestrian. In this regard, according to an embodiment of the present application, the semantic category label of the point cloud grid can be set to a pedestrian, so that the trained semantic segmentation network can accurately detect the above-mentioned pedestrians to prevent vehicles from colliding with pedestrians.

[0119] The following is combined with Figure 5 , the training of the preset semantic segmentation network and the process of semantic segmentation using the preset semantic segmentation network are briefly explained.

[0120] like Figure 5 As shown in the figure, after obtaining the laser radar point cloud, it is sequentially subjected to rasterization processing, multi-scale three-dimensional grid feature extraction, and two-dimensional grid feature extraction to obtain the two-dimensional grid features of each point cloud grid, and the two-dimensional grid features are input into the output head for semantic segmentation, and the output head will output the semantic category of each point cloud grid. In addition, after obtaining the laser radar point cloud, it is sequentially subjected to point cloud semantic truth value acquisition and two-dimensional grid semantic truth value generation to obtain grid semantic truth value, that is, the semantic category label of the point cloud grid. The loss value of the loss function is calculated based on the semantic category label and the semantic category output by the above output head, the parameter gradient of the network parameter is calculated based on the loss value, and the network parameter is updated based on the parameter gradient back propagation to complete a training. It should be noted that the two steps of point cloud semantic truth value acquisition and two-dimensional grid semantic truth value can be completed before training, and the laser radar point cloud and its grid semantic truth value can be directly used after starting training.

[0121] The method for extracting multi-scale three-dimensional grid features is the same as the method for performing downsampling of the three-dimensional grid features of the point cloud grid at multiple different scales in the aforementioned step S102 to obtain the three-dimensional sampling features of each scale; the method for extracting two-dimensional grid features is the same as the method for obtaining the two-dimensional grid features of the point cloud grid in the aforementioned step S103. The method for obtaining the semantic truth value of the point cloud is the same as the method for obtaining the semantic category of each point cloud in the point cloud grid in the aforementioned step S302; the method for generating the semantic truth value of the two-dimensional grid is the same as the method for obtaining the semantic category label of the point cloud grid in the aforementioned step S303.

[0122] The above steps S303 and S304 are further explained below.

[0123] (I) Step S303 is described.

[0124] In some implementations of step S303, the target semantic category may be acquired through the following steps S3031 to S3032.

[0125] Step S3031: From all semantic categories of the point cloud grid, a semantic category with a first priority that meets a first preset condition is selected as a target semantic category.

[0126] Those skilled in the art can flexibly set the specific content of the first preset condition according to actual needs. In some preferred implementations, the first preset condition can be selecting a semantic category with the largest first priority level.

[0127] In some embodiments, if the first priority of multiple semantic categories in all semantic categories of the point cloud grid all meet the first preset condition, then a semantic category whose number of point clouds meets the second preset condition can be selected as the target semantic category from the multiple semantic categories whose first priority all meet the first preset condition. The number of point clouds of a semantic category refers to the number of point clouds belonging to this semantic category in the current point cloud grid. Those skilled in the art can flexibly set the specific content of the second preset condition according to actual needs. In some preferred embodiments, the second preset condition is to select a semantic category with the largest number of point clouds.

[0128] Step S3032: Setting the semantic category label of the point cloud grid according to the target semantic category.

[0129] Based on the method described in steps S3031 to S3032 above, when the first priorities of multiple semantic categories all meet the first preset condition, the importance of these semantic categories can be divided again according to the number of point clouds. The larger the number of point clouds, the more important it is compared with other semantic categories that also meet the first preset condition, so that a semantic category can be finally selected to set the semantic category label of the point cloud grid.

[0130] In some implementations of step S303, when a semantic category whose point cloud quantity meets the second preset condition is selected as the target semantic category, the semantic category label of the point cloud grid can also be set through the following steps S3033 to S3035.

[0131] Step S3033: determining whether the ratio of the number of point clouds belonging to the target semantic category in the point cloud grid to all point clouds in the point cloud grid is greater than a preset ratio threshold;

[0132] If yes, it means that most of the point clouds in the point cloud grid mainly belong to the target semantic category, and the semantic category label can be directly set according to the target semantic category, so go to step S3034. If no, it means that most of the point clouds in the point cloud grid do not belong to the target semantic category, and the semantic category label can be set according to the point cloud ratio of each semantic category, so go to step S3035.

[0133] Step S3034: Set a semantic category label according to the target semantic category.

[0134] In this implementation, a one-hot encoding method may be used to encode the target semantic category, and the encoding result is used as the semantic category label.

[0135] For example, a point cloud grid contains point clouds of three semantic categories: a, b, and c, and the numbers of these three point clouds are 60, 30, and 10 respectively. The target semantic category is a, and the semantic category label obtained by the one-hot encoding method is [1 0 0].

[0136] Step S3035: setting semantic category labels according to the ratio of the number of point clouds belonging to each semantic category in all point clouds in the point cloud grid.

[0137] Continuing to refer to the example of step S3034, according to the quantity ratio of point clouds of the three semantic categories a, b, and c, the semantic category label can be set to [0.6 0.3 0.1].

[0138] Based on the method described in steps S3033 to S3035 above, the semantic category labels of the point cloud grid can be set in different ways according to the different point cloud proportions of the target semantic category, so as to more realistically reflect or characterize the semantic category of the point cloud in the point cloud grid.

[0139] (ii) Explain step S304.

[0140] In some implementations of step S304, Figure 6 The following steps S3041 to S3043 are shown to train the preset semantic segmentation network.

[0141] Step S3041: according to the second priority of each semantic category, respectively set the weight corresponding to each semantic category.

[0142] Those skilled in the art can flexibly set the weights corresponding to different semantic categories according to actual needs, and this embodiment does not specifically limit this. In some preferred embodiments, a larger weight can be set for semantic categories with a smaller number of point clouds (small categories) to improve the semantic segmentation ability of the semantic segmentation network for small categories; a larger weight can also be set for semantic categories with higher importance (important categories) to improve the semantic segmentation ability of the semantic segmentation network for important categories.

[0143] Step S3042: According to the weight corresponding to each semantic category, the loss function corresponding to each semantic category is weighted and calculated to form a target loss function.

[0144] For example, there are three semantic categories a, b, and c. The corresponding loss functions of these three semantic categories are f1, f2, and f3, and the weights are w1, w2, and w3, respectively. Through weighted sum calculation, we can get the target loss function f=w1*f1+ w2*f2 +w3*f3.

[0145] Step S3043: adopting the target loss function and training the preset semantic segmentation network according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

[0146] In this embodiment, a conventional network training method in the field of machine learning technology can be adopted, using a target loss function, and training a preset semantic segmentation network based on three-dimensional point cloud samples and semantic category labels of each point cloud grid.

[0147] Based on the method described in steps S3041 to S3043 above, different weights can be set for the loss functions of different semantic categories to adjust the proportion of different semantic categories in the loss value calculated by the target loss function, thereby achieving the adjustment of the segmentation ability of the semantic segmentation network for different semantic categories. Based on this, for the semantic category whose segmentation ability is to be improved, the weight of its loss function can be increased, and vice versa, the weight of its loss function can be reduced.

[0148] In some implementations of step S304, the loss function corresponding to each semantic category may be obtained through the following steps 11 to 12.

[0149] Step 11: For each semantic category, the cross entropy loss function and the Lovasz loss function are used to construct the first loss function and the second loss function corresponding to the semantic category respectively.

[0150] The cross entropy loss function and the Lovasz loss function are both conventional loss functions in the field of machine learning technology, and this embodiment does not describe their specific function expressions.

[0151] Step 12: For each semantic category, perform a weighted sum calculation on the first loss function and the second loss function of the semantic category to obtain the final loss function of the semantic category.

[0152] The weighted sum calculation in this step is similar to the weighted sum calculation in the aforementioned step S3042.

[0153] In the method based on the above steps 11 to 12, the cross entropy loss function can be used to directly optimize the network parameters for the probability that each point cloud grid belongs to different semantic categories (i.e., the probability distribution of the point cloud grid is optimized), and the Lovasz loss function can be used to optimize the network parameters based on the average intersection over union (IoU) index of the segmentation of all point cloud grids on a training batch. Therefore, the above method can not only reduce the difficulty of network parameter optimization, but also improve the effect of semantic segmentation.

[0154] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order, and they can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0155] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0156] Another aspect of the present application provides a computer device.

[0157] In an embodiment of a computer device according to the present application, the computer device mainly includes a storage device and a processor. The storage device can be configured to store a program for executing the point cloud semantic segmentation method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the point cloud semantic segmentation method of the above method embodiment. The computer device described in the present application may include point cloud semantic segmentation, etc. Figure 7 , Figure 7 In the example, the storage device 11 and the processor 12 are connected to each other through a bus. For the convenience of explanation, only the parts related to this embodiment are shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of this application.

[0158] In some possible implementations, the computer device may include multiple storage devices and multiple processors. The multiple processors may be processors deployed on the same device. For example, the computer device may be a high-performance device composed of multiple processors, and the multiple processors may be processors configured on the high-performance device. In addition, the multiple processors may also be processors deployed on different devices. For example, the computer device may be a server cluster, and the multiple processors may be processors on different servers in the server cluster.

[0159] Another aspect of the present application also provides a computer-readable storage medium.

[0160] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium may be configured to store a program for executing the point cloud semantic segmentation method of the above method embodiment, and the program may be loaded and run by a processor to implement the above point cloud semantic segmentation method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0161] Another aspect of the present application also provides a smart device.

[0162] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. The intelligent device described in the present application may include a driving device, a smart car, a robot, and the like. Figure 8 , Figure 8FIG. 2 exemplarily shows that the memory 21 and the processor 22 are communicatively connected via a bus.

[0163] In some embodiments of the present application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in the present application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the method described in any of the above embodiments.

[0164] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A point cloud semantic segmentation method, characterized in that: The method comprises: Performing rasterization processing on the three-dimensional point cloud to form a plurality of three-dimensional point cloud grids; For each point cloud grid, down-sampling the three-dimensional grid features of the point cloud grid at multiple different scales is performed to obtain three-dimensional sampling features at each scale, and For each point cloud grid, converting the three-dimensional sampling features of each scale of the point cloud grid into two-dimensional sampling features of each scale respectively, and fusing all the two-dimensional sampling features of the point cloud grid to form a two-dimensional grid feature of the point cloud grid; Semantic segmentation is performed according to the two-dimensional grid features of each point cloud grid to obtain the semantic category of each point cloud grid respectively; The fusing of all the two-dimensional sampling features of the point cloud grid includes: fusing the two-dimensional sampling features of each scale in order from high to low scales, so that the scale of the two-dimensional grid feature formed by fusion is the same as the scale of the two-dimensional sampling feature of the smallest scale; in, The performing semantic segmentation according to the two-dimensional grid features of each point cloud grid includes: using a preset semantic segmentation network and performing semantic segmentation according to the two-dimensional grid features of each point cloud grid; The preset semantic segmentation network is trained in the following way: Performing rasterization processing on the three-dimensional point cloud samples to form a plurality of three-dimensional point cloud grids; For each point cloud grid, respectively obtain the semantic category of each point cloud in the point cloud grid, and set the semantic category label of the point cloud grid according to the first priority of each semantic category in all semantic categories of the point cloud grid; The preset semantic segmentation network is trained according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

2. The point cloud semantic segmentation method according to claim 1, characterized in that: Before fusing all the two-dimensional sampling features of the point cloud grid, the method further includes: Obtaining a two-dimensional sampling feature of a preset scale as a target feature; A feature pyramid network is used to extract the target features to obtain new features; The new feature is used as the final two-dimensional sampling feature of the preset scale.

3. The point cloud semantic segmentation method according to claim 2, characterized in that: The feature extraction of the target features by using a feature pyramid network includes: A feature pyramid network is used to downsample the target features at multiple scales to obtain downsampled features at each scale, and According to the order of the scales from high to low, the downsampled features of each scale are fused in turn, and the fusion result is used as the new feature; The new feature has the same scale as the target feature.

4. The point cloud semantic segmentation method according to claim 1, characterized in that: The step of setting the semantic category label of the point cloud grid according to the first priority of each semantic category among all semantic categories of the point cloud grid comprises: From all semantic categories of the point cloud grid, a semantic category with a first priority that satisfies a first preset condition is selected as a target semantic category; According to the target semantic category, a semantic category label of the point cloud grid is set.

5. The point cloud semantic segmentation method according to claim 4, characterized in that: The step of selecting a semantic category with a first priority level that satisfies a first preset condition as a target semantic category includes: If the first priorities of multiple semantic categories in all semantic categories of the point cloud grid all meet the first preset condition, Then, from the multiple semantic categories whose first priorities all satisfy the first preset condition, a semantic category whose point cloud quantity satisfies the second preset condition is selected as the target semantic category.

6. The point cloud semantic segmentation method according to claim 5, characterized in that: After selecting a semantic category whose number of point clouds meets the second preset condition as the target semantic category, the method further includes setting a semantic category label of the point cloud grid in the following manner: Determine whether the number ratio of point clouds belonging to the target semantic category in the point cloud grid among all point clouds in the point cloud grid is greater than a preset ratio threshold; If yes, setting the semantic category label according to the target semantic category; If not, the semantic category label is set according to the ratio of the number of point clouds belonging to each semantic category in the point cloud grid to all point clouds in the point cloud grid.

7. The point cloud semantic segmentation method according to claim 1, characterized in that: The training of the preset semantic segmentation network includes: According to the second priority of each semantic category, a weight corresponding to each semantic category is set respectively; According to the weight corresponding to each semantic category, the loss function corresponding to each semantic category is weighted and calculated to form the target loss function; The target loss function is adopted, and the preset semantic segmentation network is trained according to the three-dimensional point cloud samples and the semantic category labels of each point cloud grid.

8. The point cloud semantic segmentation method according to claim 7, characterized in that: The loss function corresponding to each semantic category is obtained in the following way: For each semantic category, the cross entropy loss function and the Lovasz loss function are used to respectively construct the first loss function and the second loss function corresponding to the semantic category, and For each semantic category, a weighted sum calculation is performed on the first loss function and the second loss function of the semantic category to obtain a final loss function of the semantic category.

9. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the point cloud semantic segmentation method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the point cloud semantic segmentation method according to any one of claims 1 to 8.

11. A smart device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the point cloud semantic segmentation method described in any one of claims 1 to 8 is implemented.