Semantic segmentation, environment perception method, device and unmanned vehicle

Abstract

The disclosure provides a semantic segmentation, environment perception method, device and unmanned vehicle, and relates to the technical field of computer vision. The semantic segmentation method comprises the following steps: generating a two-dimensional image according to point cloud data; performing semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image; generating a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data; and performing semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data. Through the above steps, the real-time performance of the point cloud semantic segmentation processing can be improved while the accuracy of the point cloud semantic segmentation result is maintained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, in particular to the field of unmanned driving, and more particularly to a semantic segmentation and environment perception method, device and unmanned vehicle. BACKGROUND

[0002] At present, unmanned devices are used to automatically transport people or objects from one location to another. The unmanned devices collect environmental information through sensors on the devices and complete the automatic transportation. The unmanned delivery vehicles controlled based on unmanned technology greatly improve the convenience of production and life and save labor costs.

[0003] Three-dimensional environment perception technology is one of the core methods in the automatic driving technology system. This perception technology is responsible for identifying pedestrians, vehicles and other dynamic and static elements around the automatic driving vehicle to provide comprehensive environmental information to the downstream control system, and then plan the driving route, avoid static obstacles and dynamic pedestrians, vehicles, etc. In the three-dimensional environment perception technology system, the three-dimensional semantic segmentation method based on laser radar aims to identify the semantic categories of each element in the three-dimensional scene point cloud scanned by the laser radar, which is a basic task in the entire three-dimensional environment perception technology system. SUMMARY

[0004] One of the technical problems to be solved by the present disclosure is to provide a semantic segmentation and environment perception method, device and unmanned vehicle.

[0005] According to a first aspect of the present disclosure, a semantic segmentation method is provided, comprising: generating a two-dimensional image according to point cloud data; performing semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image; generating a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data; and performing semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

[0006] In some embodiments, the two-dimensional image is a depth map, and the generating a two-dimensional image according to point cloud data comprises: determining two-dimensional conversion coordinates of the point cloud data in a spherical coordinate system, wherein the two-dimensional conversion coordinates include a yaw angle and a pitch angle; assigning the point cloud data into a plurality of grids according to the two-dimensional conversion coordinates; determining a feature of each grid according to point cloud points in each grid; and constructing the depth map according to the features of all grids.

[0007] In some embodiments, the semantic segmentation result corresponding to the two-dimensional image comprises semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image, and the generating the point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data comprises: determining a point cloud point in the point cloud data that matches each point in the two-dimensional image; splicing the semantic label information corresponding to each point in the two-dimensional image, the feature representation corresponding to each point in the two-dimensional image, and coordinates of the matched point cloud point to obtain spliced feature representation corresponding to each point in the two-dimensional image; and constructing the point cloud feature map according to the spliced feature representation corresponding to each point in the two-dimensional image.

[0008] In some embodiments, the semantic segmentation of the two-dimensional image based on the first neural network model comprises: feature extraction of the two-dimensional image based on an encoder module, and output of a feature map obtained by the feature extraction to a decoder module; and decoding of the feature map based on the decoder module to obtain the semantic segmentation result corresponding to the two-dimensional image.

[0009] In some embodiments, the feature extraction of the two-dimensional image based on the encoder module and the output of the obtained feature map to the decoder module comprise: feature extraction of the two-dimensional image based on a first encoding unit, and output of a feature map obtained by the feature extraction of the first encoding unit to a second encoding unit and the decoder module; and feature extraction of the feature map output by the first encoding unit based on the second encoding unit, and output of a feature map obtained by the feature extraction of the second encoding unit to the decoder module, wherein the first encoding unit and the second encoding unit are different in structure.

[0010] In some embodiments, at least one of the first encoding unit and the second encoding unit comprises a plurality of convolution layers, and at least one of the plurality of convolution layers uses a long strip-shaped convolution kernel.

[0011] In some embodiments, the first encoding unit comprises first to third convolution layers arranged in sequence from an input side to an output side, wherein the first and third convolution layers use square convolution kernels, and the second convolution layer uses a long strip-shaped convolution kernel.

[0012] In some embodiments, the decoder module comprises a plurality of decoding units arranged in sequence from an input side to an output side, and a semantic label classification layer, and the decoding unit comprises an up-sampling layer and a plurality of convolution layers.

[0013] In some embodiments, the second neural network model comprises a plurality of convolution layers and a semantic label classification layer, wherein the convolution layers use 1*1 convolution kernels.

[0014] According to a second aspect of the present disclosure, an environment perception method is provided, comprising: obtaining point cloud data collected by an unmanned vehicle; determining semantic label information of the point cloud data according to the semantic segmentation method described above; and determining environment information in which the unmanned vehicle is located according to the semantic label information of the point cloud data.

[0015] According to a third aspect of the present disclosure, a semantic segmentation device is provided, comprising: a first generation module configured to generate a two-dimensional image according to point cloud data; a first segmentation module configured to perform semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image; a second generation module configured to generate a point cloud feature map according to the semantic segmentation result of the two-dimensional image and the point cloud data; and a second segmentation module configured to perform semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

[0016] In some embodiments, the two-dimensional image is a depth map, and the first generation module is configured to: determine two-dimensional conversion coordinates of the point cloud data in a spherical coordinate system, wherein the two-dimensional conversion coordinates include a yaw angle and a pitch angle; distribute the point cloud data into a plurality of grids according to the two-dimensional conversion coordinates; determine a feature of each grid according to point cloud points in each grid; and construct the depth map according to the features of all grids.

[0017] In some embodiments, the second generation module is configured to: determine a point cloud point in the point cloud data that matches each point in the two-dimensional image; splice semantic label information corresponding to each point in the two-dimensional image and a feature representation corresponding to each point in the two-dimensional image, and coordinates of the matching point cloud point, to obtain a spliced feature representation corresponding to each point in the two-dimensional image; and construct the point cloud feature map according to the spliced feature representation corresponding to each point in the two-dimensional image.

[0018] According to a fourth aspect of the present disclosure, another semantic segmentation device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the semantic segmentation method described above based on instructions stored in the memory.

[0019] According to a fifth aspect of the present disclosure, an environment perception device is provided, comprising: an obtaining module configured to obtain point cloud data collected by an unmanned vehicle; a semantic segmentation device as described above; and a determining module configured to determine environment information in which the unmanned vehicle is located according to semantic label information of the point cloud data.

[0020] According to a sixth aspect of the present disclosure, an environment perception device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the environment perception method as described above based on instructions stored in the memory.

[0021] According to a seventh aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon computer program instructions which, when executed by a processor, implement the semantic segmentation method as described above or the environment perception method as described above.

[0022] According to an eighth aspect of the present disclosure, an unmanned vehicle is also provided, comprising the semantic segmentation apparatus or the environment perception device as described above.

[0023] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0025] The present disclosure can be understood more fully by reference to the following detailed description of exemplary embodiments thereof, which are illustrated in the accompanying drawings.

[0026] Figure 1 A flowchart of a semantic segmentation method according to some embodiments of the present disclosure;

[0027] Figure 2 A flowchart of generating a two-dimensional image from point cloud data according to some embodiments of the present disclosure;

[0028] Figure 3 A flowchart of performing semantic segmentation based on a first neural network model according to some embodiments of the present disclosure;

[0029] Figure 4a A structure diagram of a first neural network model according to some embodiments of the present disclosure;

[0030] Figure 4b A structure diagram of a first encoding unit according to some embodiments of the present disclosure;

[0031] Figure 4c A structure diagram of a second encoding unit according to some embodiments of the present disclosure;

[0032] Figure 4d A structure diagram of a decoding unit according to some embodiments of the present disclosure;

[0033] Figure 5 A flowchart of an environment perception method according to some embodiments of the present disclosure;

[0034] Figure 6 A structural schematic diagram of a semantic segmentation device according to some embodiments of the present disclosure;

[0035] Figure 7 A structural schematic diagram of an environment perception device according to some embodiments of the present disclosure;

[0036] Figure 8 A structural schematic diagram of a semantic segmentation device or an environment perception device according to some embodiments of the present disclosure;

[0037] Figure 9 A structural schematic diagram of a computer system according to some embodiments of the present disclosure;

[0038] Figure 10 A structural schematic diagram of an unmanned vehicle according to some embodiments of the present disclosure;

[0039] Figure 11 A perspective structural schematic diagram of an unmanned vehicle according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0040] Various exemplary embodiments of the present disclosure will now be described in detail herein below with reference to the drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless specifically stated otherwise.

[0041] Meanwhile, it should be understood that the sizes of the various portions shown in the drawings are not drawn in proportion to the actual proportions.

[0042] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the present disclosure or its application or uses.

[0043] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the description of the present disclosure where appropriate.

[0044] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0045] Note that similar reference numerals and letters refer to similar items throughout the drawings, and thus, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0046] In order to make the purposes, technical solutions, and advantages of the present disclosure clearer, the present disclosure is further described in detail below with reference to specific embodiments and with reference to the drawings.

[0047] In the related art, in order to meet the real-time requirement of engineering application, the three-dimensional point cloud semantic segmentation method is usually based on the principle of spherical projection, converts the three-dimensional laser radar point cloud into a depth map (Range View), and then projects the obtained semantic label information back into the original three-dimensional laser radar point cloud after performing semantic segmentation on the depth map by means of a two-dimensional convolutional neural network.

[0048] The main problem of the above method is that in the process of projecting the three-dimensional point cloud data into a depth map, information loss usually occurs, for example, projecting several three-dimensional point cloud points into the same pixel in the depth map will cause the loss of distinction between these three-dimensional point cloud points falling in the same pixel, thereby reducing the accuracy of the segmentation result.

[0049] To solve this problem, the related art often increases the resolution of the depth map to minimize the number of points projected into the same pixel and improve the distinction after point cloud projection. However, this will increase the computational resource overhead and negatively affect the real-time performance of the three-dimensional point cloud semantic segmentation method.

[0050] Figure 1 A flowchart of a semantic segmentation method according to some embodiments of the present disclosure is shown. As shown in Figure 1 The semantic segmentation method of the embodiments of the present disclosure includes:

[0051] Step S110: generating a two-dimensional image according to point cloud data.

[0052] In some embodiments, the two-dimensional image is a depth map (Range View), and in step S110, a depth map is generated according to the point cloud data.

[0053] In other embodiments, the two-dimensional image is a bird's eye view (Bird's Eye View, BEV), and in step S110, a bird's eye view is generated according to the point cloud data.

[0054] In some embodiments, the point cloud data is point cloud data collected by a laser radar. For example, during the driving of an unmanned vehicle, point cloud data is collected by a laser radar installed on the unmanned vehicle.

[0055] Step S120: performing semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image.

[0056] In some embodiments, the first neural network model is a deep neural network model, which includes an encoder module and a decoder module.

[0057] In some embodiments, the semantic segmentation result of the two-dimensional image comprises semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image. Illustratively, in an autonomous driving scenario, the semantic label information comprises label information of pedestrians, vehicles, lanes, sidewalks, and the like. Illustratively, the feature representation corresponding to each point in the two-dimensional image is a feature vector output after processing by the first neural network model.

[0058] Step S130: generating a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data.

[0059] In some embodiments, the semantic segmentation result of the two-dimensional image comprises semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image. In these embodiments, step S130 comprises steps S131 to S133.

[0060] Step S131: determining a point cloud point in the point cloud data that matches each point in the two-dimensional image.

[0061] In some embodiments, after the two-dimensional image is generated by step S110, a mapping relationship between points in the two-dimensional image and point cloud points in the point cloud data is saved. In step S131, according to the mapping relationship, a point cloud point in the point cloud data that matches each point in the two-dimensional image is determined.

[0062] Step S132: concatenating the semantic label information corresponding to each point in the two-dimensional image and the feature representation corresponding to each point in the two-dimensional image, and the coordinates of the matched point cloud point, to obtain a concatenated feature representation corresponding to each point in the two-dimensional image.

[0063] In some embodiments, the concatenation is performed in the following order: the semantic label information corresponding to each point in the two-dimensional image, the feature representation corresponding to each point in the two-dimensional image, and the coordinates of the matched point cloud point, to obtain the concatenated feature representation.

[0064] For example, assume that the semantic label information corresponding to any point in the two-dimensional image is represented by vector A ij , the feature representation corresponding to the point is represented by vector B ij , the coordinates of the point cloud point matching the point are represented by vector C ij , the concatenated feature representation is represented by vector D ij , and D ij = (Aij, Bij, C ij ).

[0065] In some embodiments, the semantic label information corresponding to each point in the two-dimensional image, the feature representation corresponding to each point in the two-dimensional image, and the coordinates of the matched point cloud points are spliced in other splicing orders, so as to obtain the spliced feature representation

[0066] Step S133: constructing a point cloud feature map according to the spliced feature representation corresponding to each point in the two-dimensional image.

[0067] In some embodiments, the entire spliced feature representation corresponding to all points in the two-dimensional image is taken as the point cloud feature map. Exemplarily, the point cloud feature map can be represented in the form of a matrix.

[0068] Step S140: performing semantic segmentation processing on the point cloud feature map based on a second neural network model, to obtain semantic label information of the point cloud data.

[0069] In some embodiments, the second neural network model includes a plurality of convolution layers and a semantic label classification layer, wherein the convolution layers adopt a convolution kernel with a size of 1*1. Compared with the structure of the first neural network model, the structure of the second neural network model is more lightweight.

[0070] In some other embodiments, the second neural network model includes a first convolution layer, a batch normalization (BN) layer, an activation function layer, a second convolution layer, and a semantic label classification layer arranged in sequence from the input end to the output end. The first convolution layer and the second convolution layer each adopt a convolution kernel with a size of 1*1; the activation function layer adopts a Relu function; and the semantic label classification layer adopts a Softmax function.

[0071] In some other embodiments, the second neural network model can also adopt other network structures capable of realizing the function of semantic segmentation of point cloud.

[0072] In the embodiments of the present disclosure, by generating a point cloud feature map according to the semantic segmentation result and the point cloud data after performing semantic segmentation on the two-dimensional image generated based on the first neural network model, and performing semantic segmentation on the point cloud feature map based on the second neural network, the processing time consumption can be reduced while maintaining the accuracy of semantic segmentation of point cloud, the real-time performance of semantic segmentation of point cloud is improved, and the requirements of safety and real-time performance of automatic driving are met.

[0073] Figure 2 A flowchart of a process of generating a two-dimensional image according to some embodiments of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the process of generating a two-dimensional image according to some embodiments of the present disclosure includes the following steps. Figure 2

[0074] Step S111: determining a two-dimensional conversion coordinate of the point cloud data in the spherical coordinate system.​

[0075] In some embodiments, the point cloud data is laser radar collected point cloud data. Exemplarily, the laser radar collected point cloud data includes three-dimensional coordinates (x, y, z) of point cloud points, and attribute data of the point cloud points, such as remission and depth of the point cloud points.

[0076] In step S111, two-dimensional conversion coordinates of the point in spherical coordinates are calculated according to the three-dimensional coordinates of the point cloud points. The two-dimensional conversion coordinates include a yaw angle and a pitch angle, and the definitions of the yaw angle and the pitch angle are as follows:

[0077]

[0078]

[0079] Wherein, yaw is the yaw angle, pitch is the pitch angle, x is the coordinate of the point cloud point on the X axis, and y is the coordinate of the point cloud point on the Y axis.

[0080] Step S112: According to the two-dimensional conversion coordinates, the point cloud data is distributed into a plurality of grids.

[0081] In some embodiments, a plurality of grids are set according to the value range of the two-dimensional conversion coordinates corresponding to the point cloud data. For example, the number of horizontal grids is determined according to the value range of the yaw angle corresponding to the point cloud data and the horizontal size of each grid, and the number of vertical grids is determined according to the value range of the pitch angle corresponding to the point cloud data and the vertical size of each grid, and the formula is as follows:

[0082]

[0083]

[0084] Wherein, n is the number of horizontal grids, m is the number of vertical grids, max yaw is the maximum yaw angle corresponding to the point cloud data, min yaw is the minimum yaw angle corresponding to the point cloud data, weight is the horizontal size of each grid, max pitch is the maximum pitch angle corresponding to the point cloud data, min pitch is the minimum pitch angle corresponding to the point cloud data, height is the vertical size of each grid.

[0085] After setting a plurality of grids, the point cloud data is projected into each grid according to the two-dimensional conversion coordinates of the point cloud data and the size of the grid.

[0086] Step S113: According to the point cloud points in each grid, the features of each grid are determined.

[0087] In some embodiments, the feature of each grid is taken as the feature of the grid, which is the point cloud point with the minimum depth to the origin. Wherein, the feature of the grid can be represented in the form of a feature vector. For example, the three-dimensional coordinates x, y, z, and the reflectivity remission, and the distance depth of the point cloud point with the minimum depth to the origin are spliced to obtain the feature vector (x, y, z, remission, depth) of the grid.

[0088] Step S114: constructing a depth map according to the features of all grids.

[0089] In some embodiments, the feature matrix composed of the feature vectors of all grids is taken as the depth map.

[0090] In the embodiments of the present disclosure, the point cloud data can be mapped into a depth map through the above steps, which facilitates subsequent semantic segmentation based on the depth map. Compared with the method of mapping point cloud data into a BEV map, the processing speed of three-dimensional point cloud data semantic segmentation can be further improved to meet the real-time requirements of the autonomous driving scene.

[0091] Figure 3 A flowchart of a semantic segmentation process based on a first neural network model according to some embodiments of the present disclosure is shown. As shown in Figure 3 the flowchart of the semantic segmentation process based on the first neural network model according to the embodiments of the present disclosure includes:

[0092] Step S131: performing feature extraction on the two-dimensional image based on the encoder module, and outputting the obtained feature map to the decoder module.

[0093] The first semantic segmentation network model includes an encoder module and a decoder module. In some embodiments, the network structure of the first semantic segmentation network model is based on the large model design idea of ConvNeXt (a network model). In order to meet the real-time requirements of autonomous driving, the network structure is miniaturized and improved, for example, the number of internal network layers is pruned, and a long strip-shaped convolution kernel is designed according to the characteristics of the depth map, which makes the processing speed faster than ConvNeXt and meets the accuracy requirements of unmanned vehicles.

[0094] In some embodiments, the encoder module includes a first encoding unit and a second encoding unit with different structures, wherein the first encoding unit is mainly used for extracting bottom layer features, and the second encoding unit is mainly used for extracting high layer features. In these embodiments, step S131 includes step S1311 and step S1312.

[0095] Step S1311: feature extraction is performed on the two-dimensional image based on the first encoding unit, and a feature map obtained by the first encoding unit is output to the second encoding unit and the decoder module.

[0096] Step S1312: feature extraction is performed on the feature map output by the first encoding unit based on the second encoding unit, and a feature map obtained by the second encoding unit is output to the decoder module.

[0097] In some embodiments, the first encoding unit includes a plurality of convolutional layers, at least one of which uses a long strip-shaped convolutional kernel.

[0098] In some embodiments, the second encoding unit includes a plurality of convolutional layers, at least one of which uses a long strip-shaped convolutional kernel.

[0099] Wherein, the long strip-shaped convolutional kernel refers to a convolutional kernel with a horizontal size not equal to a vertical size. For example, a convolutional kernel with a size of 5*9 is a long strip-shaped convolutional kernel, and a 1*1 convolutional kernel is a square convolutional kernel.

[0100] Step S132: decoding is performed on the feature map based on the decoder module to obtain a semantic segmentation result corresponding to the two-dimensional image.

[0101] In some embodiments, the decoder module includes a plurality of decoding units arranged in sequence from the input side to the output side, and a semantic label classification layer. In these embodiments, the input feature map is decoded layer by layer by the plurality of decoding units, and then semantic label prediction is performed by the semantic label classification layer.

[0102] In the embodiments of the present disclosure, through the above steps, the depth map can be quickly and semantically segmented based on the first neural network model, meeting the real-time requirements of the autonomous driving scenario. Further, by projecting the semantic segmentation result of the depth map to the point cloud data and constructing a point cloud feature map therefrom, and performing semantic segmentation on the point cloud feature map based on the second neural network model, the accuracy of point cloud semantic segmentation can be improved while meeting the real-time requirements.

[0103] Figure 4a A structural schematic diagram of the first neural network model according to some embodiments of the present disclosure is shown. As shown in FIG. 4, the first neural network model of the embodiments of the present disclosure includes an encoder module 410 and a decoder module 420. Figure 4a

[0104] The encoder module 410 includes a first encoding unit 411 and a second encoding unit 412. The decoder module 420 includes a plurality of decoding units 421 and a semantic label classification layer (not shown in the figure). Exemplarily, the semantic label classification layer can be composed of a single convolutional layer.

[0105] ​In some embodiments, the encoder module 410 includes a first encoding unit 411 and three second encoding units 412 arranged sequentially from the input side to the output side; the decoder module includes... For ease of explanation, ... Figure 4a The three second coding units from top to bottom are called coding unit e1, coding unit e2, and coding unit e3. Figure 4a The three decoding units from top to bottom are called decoding unit d1, decoding unit d2, and decoding unit d3.

[0106] The first encoding unit 411 extracts features from the input depth map and then processes the obtained features. Figure 1 The output is sent to encoding unit e1 and decoding unit d1. Encoding unit e1 processes the input features... Figure 1 Perform feature extraction and obtain the features Figure 2 The output is sent to encoding unit e2 and decoding unit d2. Encoding unit e2 processes the input features. Figure 2 Perform feature extraction and obtain the features Figure 3 The output is sent to encoding unit e3 and decoding unit d3. Encoding unit e3 processes the input features... Figure 3 Feature extraction is performed, and the resulting feature map 4 is output to the decoding unit d3.

[0107] Decoding unit d3 for input features Figure 3 Decode the feature map 4 and the resulting features Figure 5 Output to decoding unit d2; decoding unit d2 processes the input features Figure 2 and characteristics Figure 5 Perform decoding processing and extract the features. Figure 6 The output is sent to the decoding unit d1; the decoding unit d1 processes the input features. Figure 1 and characteristics Figure 6 Perform decoding processing and extract the features. Figure 7 The output is fed to the semantic label classification layer to obtain the semantic label information corresponding to the depth map, and the features are then processed. Figure 7 This is the feature map that is the final output of the first neural network model.

[0108] In some embodiments, the input depth map has a resolution of W*H and a feature dimension of 5. The resolution and output feature dimension of the feature maps output by each unit in the encoder and decoder modules satisfy the following: the resolution of the feature map output by the first encoding unit is W*H, and the output feature dimension is 32; the resolution of the feature map output by the first second encoding unit (i.e., encoding unit e1) is... The output feature dimension is 32-dimensional, and the resolution of the feature map output by the second coding unit (i.e., coding unit e2) is... The output feature dimension is 64, and the resolution of the feature map output by the third second encoding unit (i.e., encoding unit e3) is The output feature dimension is 128, and the resolution of the feature map output by the first decoding unit (i.e., decoding unit d3) is The output feature dimension is 64, and the resolution of the feature map output by the second decoding unit (i.e., decoding unit d2) is The output feature dimension is 32, and the resolution of the feature map output by the third decoding unit (i.e., decoding unit d1) is W*H, and the output feature dimension is 32. Wherein, W and H are integers greater than 1, for example, W is 16 and H is 640.

[0109] Figure 4b A structural schematic diagram of the first encoding unit according to some embodiments of the present disclosure is shown in FIG. 4. As shown in FIG. 4, the first encoding unit 411 of the embodiments of the present disclosure includes a first convolutional layer 4111, a second convolutional layer 4112, and a third convolutional layer 4113. Figure 4b

[0110] In some embodiments, the first convolutional layer 4111 adopts a convolutional kernel with a size of 1*1, the second convolutional layer 4112 adopts a convolutional kernel with a size of 5*9, and the third convolutional layer adopts a convolutional kernel with a size of 1*1. Wherein, the second convolutional layer adopts a depth-wise convolutional manner. One convolutional kernel in the depth-wise convolutional manner is responsible for one channel of the input picture, and one channel of the input picture is only convolved by one convolutional kernel.

[0111] In some embodiments, a Relu (Relu is an activation function) layer is further arranged between the first convolutional layer and the second convolutional layer, a batch normalization layer is further arranged between the second convolutional layer and the second convolutional layer, and a batch normalization layer and a Relu layer are further arranged after the third convolutional layer.

[0112] In the embodiments of the present disclosure, by adopting the combination structure of the convolutional layer based on the 1*1 convolutional kernel, the convolutional layer with the 5*9 depth convolutional kernel, and the convolutional layer with the 1*1 convolutional kernel, compared with the structure of adopting a single layer of large convolutional kernel, the processing speed can be improved without losing the perception range and accuracy.

[0113] Figure 4c A structural schematic diagram of the second encoding unit according to some embodiments of the present disclosure is shown in FIG. 5. As shown in FIG. 5, the second encoding unit 412 of the embodiments of the present disclosure includes a first convolutional layer 4121, a second convolutional layer 4122, a third convolutional layer 4123, a fourth convolutional layer 4124, and an average pooling layer 4124. Figure 4c

[0114] ​​In some embodiments, the first convolutional layer 4121 adopts a convolutional kernel with a size of 1*1, the second convolutional layer 4122 adopts a convolutional kernel with a size of 3*11, the third convolutional layer 4123 adopts a convolutional kernel with a size of 1*1, and the fourth convolutional layer 4124 adopts a convolutional kernel with a size of 1*1.

[0115] In some embodiments, a Relu layer is further arranged between the first convolutional layer 4121 and the second convolutional layer 4122, a batch normalization layer is further arranged between the second convolutional layer 4122 and the third convolutional layer 4123, a Relu layer is further arranged between the third convolutional layer 4123 and the fourth convolutional layer 4124, and a batch normalization layer and a Relu layer are further arranged between the fourth convolutional layer 4124 and the average pooling layer 4125. The second convolutional layer 4122 adopts a depth-wise convolution manner.

[0116] In the embodiments of the present disclosure, by adopting a long strip-shaped convolutional kernel in the second convolutional layer of the second encoding unit, the size of the depth map can be matched, and by additionally arranging a ReLU activation function after the fourth convolutional layer, the model effect can be improved and the time consumption can be reduced.

[0117] Figure 4d FIG. 4 is a structural schematic diagram of a decoding unit according to some embodiments of the present disclosure. As shown in FIG. 4, the decoding unit 421 of the embodiments of the present disclosure includes an up-sampling layer 4211, a first convolutional layer 4212, a second convolutional layer 4213, a third convolutional layer 4214, and a semantic label classification layer 4215. Figure 4d

[0118] In some embodiments, the up-sampling layer 4211 is a PixelShuffle layer. PixelShuffle is an up-sampling method, and its main function is to obtain a high-resolution feature map from a low-resolution feature map through convolution and multi-channel reorganization. PixelShuffle can effectively enlarge the reduced feature map, and can replace the interpolation or deconvolution method to realize up-sampling.

[0119] In some embodiments, the first convolutional layer 4212 adopts a convolutional kernel with a size of 3*3, the second convolutional layer 4213 adopts a convolutional kernel with a size of 1*1, and the third convolutional layer adopts a convolutional layer with a size of 1*1.

[0120] In the embodiments of the present disclosure, by introducing the PixelShuffle layer in the decoding unit, the low-resolution feature can be restored to a high-resolution feature, which helps to improve the semantic segmentation accuracy. In addition, by adopting a small-size convolutional kernel in the decoding unit, the calculation complexity can be reduced and the redundancy of the same perception range can be reduced, which helps to improve the processing speed. ​

[0121] Figure 5 A flowchart of an environment perception method according to some embodiments of the present disclosure is shown in FIG. 5. As shown in FIG. 5, the environment perception method of the embodiments of the present disclosure includes the following steps. Figure 5

[0122] Step S510: Obtain point cloud data collected by an unmanned vehicle.

[0123] In some embodiments, the point cloud data is collected by a laser radar installed on the unmanned vehicle, and the point cloud data collected by the laser radar is transmitted to an environment perception device.

[0124] Step S520: Generate a two-dimensional image according to the point cloud data.

[0125] In some embodiments, the point cloud data is mapped in the manner shown in FIG. 6 to obtain a depth map. Figure 2

[0126] Step S530: Perform semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image.

[0127] Step S540: Generate a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data.

[0128] Step S550: Perform semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

[0129] Step S560: Determine environment information in which the unmanned vehicle is located according to the semantic label information of the point cloud data.

[0130] For example, if the semantic label information of the point cloud data determined through steps S520 to S560 includes a pedestrian class label and a vehicle class label, it can be determined that the environment in which the unmanned vehicle is located includes pedestrians and vehicles. Next, according to the environment information in which the unmanned vehicle is located, each type of dynamic and static element in the scene can be further subdivided and identified, such as further subdividing and identifying vehicles.

[0131] In the embodiments of the present disclosure, the above steps are used to accurately and in real time perceive the environment in which the unmanned vehicle is located, which can meet the requirements of safety and real-time performance of autonomous driving.

[0132] Figure 6 A structural diagram of a semantic segmentation device according to some embodiments of the present disclosure is shown in FIG. 6. As shown in FIG. 6, the semantic segmentation device 600 of the embodiments of the present disclosure includes a first generation module 610, a first segmentation module 620, a second generation module 630, and a second segmentation module 640. Figure 6

[0133] ​​​The first generation module 610 is configured to generate a two-dimensional image according to the point cloud data.

[0134] In some embodiments, the two-dimensional image is a range view, and the first generation module 610 generates the range view according to the point cloud data.

[0135] In other embodiments, the two-dimensional image is a bird’s eye view (BEV), and the first generation module 610 generates the bird’s eye view according to the point cloud data.

[0136] In some embodiments, the first generation module 610 generating the range view according to the point cloud data includes: the first generation module 610 determining two-dimensional conversion coordinates of the point cloud data in a spherical coordinate system, wherein the two-dimensional conversion coordinates include a yaw angle and a pitch angle; the first generation module 610 assigning the point cloud data into a plurality of grids according to the two-dimensional conversion coordinates; the first generation module 610 determining a feature of each grid according to point cloud points in each grid; and the first generation module 610 constructing the range view according to the features of all the grids.

[0137] The first segmentation module 620 is configured to perform semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image.

[0138] In some embodiments, the first neural network model includes an encoder module and a decoder module. In these embodiments, the first segmentation module 620 is configured to: perform feature extraction on the two-dimensional image based on the encoder module, and output a feature map obtained to the decoder module; and perform decoding on the feature map based on the decoder module to obtain the semantic segmentation result corresponding to the two-dimensional image.

[0139] The second generation module 630 is configured to generate a point cloud feature map according to the semantic segmentation result of the two-dimensional image and the point cloud data.

[0140] In some embodiments, the semantic segmentation result of the two-dimensional image includes semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image. In these embodiments, the second generation module 630 generating the point cloud feature map according to the semantic segmentation result of the two-dimensional image and the point cloud data includes: the second generation module 630 determining point cloud points in the point cloud data that match each point in the two-dimensional image; the second generation module 630 concatenating the semantic label information corresponding to each point in the two-dimensional image and the feature representation corresponding to each point in the two-dimensional image, and coordinates of the matched point cloud points, to obtain a concatenated feature representation corresponding to each point in the two-dimensional image; and the second generation module 630 constructing the point cloud feature map according to the concatenated feature representation corresponding to each point in the two-dimensional image.

[0141] The second segmentation module 640 is configured to perform semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

[0142] In the embodiments of the present disclosure, the device can reduce the processing time consumption while maintaining the accuracy of point cloud semantic segmentation, improve the real-time performance of point cloud semantic segmentation, and meet the requirements of safety and real-time performance of autonomous driving.

[0143] Figure 7 FIG. 1 is a structural schematic diagram of an environment perception device according to some embodiments of the present disclosure. As shown in FIG. 1, the environment perception device 100 according to some embodiments of the present disclosure includes an acquisition module 110, a semantic segmentation device 120, and a determination module 130. Figure 7 The acquisition module 110 is configured to acquire point cloud data collected by an unmanned vehicle.

[0144] The semantic segmentation device 120 is configured to generate a two-dimensional image according to the point cloud data; perform semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image; generate a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data; and perform semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

[0145] The determination module 130 is configured to determine environment information in which the unmanned vehicle is located according to the semantic label information of the point cloud data.

[0146] In the embodiments of the present disclosure, the device can accurately and in real time perceive the environment in which the unmanned vehicle is located, and can meet the requirements of safety and real-time performance of autonomous driving.

[0147]

[0148] FIG. 2 is a structural schematic diagram of a semantic segmentation device or an environment perception device according to some embodiments of the present disclosure. Figure 8 As shown in FIG. 2, the semantic segmentation device or the environment perception device 200 includes a memory 210 and a processor 220 coupled to the memory 210. The memory 210 is configured to store instructions for implementing any of the embodiments of the semantic segmentation method or the environment perception method. The processor 220 is configured to execute the semantic segmentation method or the environment perception method according to any of the embodiments of the present disclosure based on the instructions stored in the memory 210.

[0149] Figure 8 FIG. 3 is a structural schematic diagram of a computer system according to some embodiments of the present disclosure.

[0150] Figure 9 As shown in FIG. 3, the computer system 300 includes a memory 310 and a processor 320 coupled to the memory 310. The memory 310 is configured to store instructions for implementing any of the embodiments of the semantic segmentation method or the environment perception method. The processor 320 is configured to execute the semantic segmentation method or the environment perception method according to any of the embodiments of the present disclosure based on the instructions stored in the memory 310.

[0151] As shown in FIG. 3, the computer system 300 includes a memory 310 and a processor 320 coupled to the memory 310. The memory 310 is configured to store instructions for implementing any of the embodiments of the semantic segmentation method or the environment perception method. The processor 320 is configured to execute the semantic segmentation method or the environment perception method according to any of the embodiments of the present disclosure based on the instructions stored in the memory 310. Figure 9 ​As shown, the computer system 900 can be in the form of a general- purpose computing device. The computer system 900 includes a memory 910, a processor 920, and a bus 930 that couples the different system components.

[0152] The memory 910 can include, for example, system memory, non-volatile storage media, and the like. The system memory stores, for example, an operating system, application programs, a Boot Loader, and other programs such as instructions for performing at least one of a semantic segmentation method or an environment perception method. The system memory can include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media stores, for example, instructions for performing a corresponding embodiment of at least one of a semantic segmentation method or an environment perception method. The non-volatile storage media includes, but is not limited to, magnetic storage media, optical storage media, flash memory, and the like.

[0153] The processor 920 can be implemented with a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field- programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and the like discrete hardware components. Accordingly, each module such as the first generation module, the first segmentation module can be implemented by a central processing unit (CPU) running instructions in a memory for performing corresponding steps, or by a dedicated circuit for performing corresponding steps.

[0154] The bus 930 can use any of a variety of bus structures. For example, the bus structure includes, but is not limited to, an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus.

[0155] The computer system 900 can be connected between the interfaces 940, 950, 960 and the memory 910 and the processor 920 through the bus 930. The input / output interface 940 can provide a connection interface for display, mouse, keyboard, and the like input / output devices. The network interface 950 provides a connection interface for various networking devices. The storage interface 960 provides a connection interface for external storage devices such as floppy disks, U disks, SD cards, and the like.

[0156] Figure 10 A structural schematic diagram of an unmanned vehicle according to some embodiments of the present disclosure; Figure 11 A perspective view of an unmanned vehicle according to some embodiments of the present disclosure. The following will be described in combination with Figure 10 and Figure 11 An unmanned vehicle provided by embodiments of the present disclosure is described.

[0157] As shown in the accompanying Figure 10 The unmanned vehicle includes a chassis module 1010, an autonomous driving module 1020, a cargo box module 1030, and a remote monitoring push module 1040.

[0158] In some embodiments, the chassis module 1010 mainly includes a battery, a power management device, a chassis controller, a motor driver, and a power motor. The battery provides power for the entire unmanned vehicle system, the power management device converts the battery output into different voltage levels available for each functional module, and controls power-on and power-off. The chassis controller accepts the motion instructions issued by the automatic driving module, and controls the unmanned vehicle to turn, move forward, move backward, brake, and the like.

[0159] In some embodiments, the automatic driving module 1020 includes a core processing unit (Orin or Xavier module), a traffic light recognition camera, front and rear surround view cameras, a multi-line laser radar, a positioning module (such as Beidou, GPS, etc.), and an inertial navigation unit. The cameras can communicate with the automatic driving module. In order to improve transmission speed and reduce wiring, GMSL link communication can be used.

[0160] In some embodiments, the automatic driving module 1020 includes the semantic segmentation device or the environment perception device in the above embodiments.

[0161] In some embodiments, the remote monitoring push module 1030 is composed of a front monitoring camera, a rear monitoring camera, a left monitoring camera, a right monitoring camera, and a push module. The module transmits video data collected by the monitoring cameras to a background server for viewing by background operators. The wireless communication module communicates with the background server through an antenna, and can realize remote control of the unmanned vehicle by the background operators.

[0162] The cargo box module 1040 is a cargo carrying device of the unmanned vehicle. In some embodiments, the cargo box module 1040 is further provided with a display interaction module. The display interaction module is used for interaction between the unmanned vehicle and the user. The user can perform operations such as picking up goods, storing goods, and purchasing goods through the display interaction module. The type of the cargo box can be changed according to actual needs. For example, in a logistics scenario, the cargo box can include multiple sub-boxes of different sizes, which can be used to load goods for distribution. In a retail scenario, the cargo box can be set as a transparent box to allow the user to directly view the products for sale.

[0163] The unmanned vehicle of the embodiments of the present disclosure can maintain the accuracy of the point cloud semantic segmentation result while improving the real-time performance of the point cloud semantic segmentation processing, thereby meeting the requirements of safety and real-time performance of automatic driving.

[0164] Here, various aspects of the disclosure are described with reference to flowcharts and / or block diagrams of methods, apparatuses and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0165] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart block or blocks or in the block or blocks of the corresponding figure.

[0166] These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable storage medium produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks or in the block or blocks of the corresponding figure.

[0167] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both software and hardware aspects.

[0168] Through the semantic segmentation, environment perception method, device and unmanned vehicle in the above embodiments, the real-time performance of point cloud semantic segmentation processing can be improved while maintaining the accuracy of the point cloud semantic segmentation result, thereby meeting the requirements of safety and real-time performance of automatic driving.

[0169] So far, the semantic segmentation, environment perception method, device and unmanned vehicle according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

Claims

1. A semantic segmentation method, comprising: generating a two-dimensional image according to point cloud data; performing semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image, the semantic segmentation result corresponding to the two-dimensional image comprising semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image; generating a point cloud feature map according to the semantic segmentation result corresponding to the two-dimensional image and the point cloud data, comprising: determining a point cloud point in the point cloud data that matches each point in the two-dimensional image; concatenating the semantic label information corresponding to each point in the two-dimensional image, the feature representation corresponding to each point in the two-dimensional image, and coordinates of the matched point cloud point to obtain concatenated feature representation corresponding to each point in the two-dimensional image; and constructing the point cloud feature map according to the concatenated feature representation corresponding to each point in the two-dimensional image; performing semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

2. The semantic segmentation method of claim 1, wherein, The two-dimensional image is a depth map, and the generating a two-dimensional image according to point cloud data comprises: determining two-dimensional conversion coordinates of the point cloud data in a spherical coordinate system, wherein the two-dimensional conversion coordinates comprise a yaw angle and a pitch angle; distributing the point cloud data into a plurality of grids according to the two-dimensional conversion coordinates; determining a feature of each grid according to point cloud points in each grid; constructing the depth map according to the features of all grids.

3. The semantic segmentation method of claim 1, wherein, The performing semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image comprises: extracting features of the two-dimensional image based on an encoder module, and outputting a feature map obtained to a decoder module; decoding the feature map based on the decoder module to obtain the semantic segmentation result corresponding to the two-dimensional image.

4. The semantic segmentation method of claim 3, wherein, The extracting features of the two-dimensional image based on an encoder module, and outputting a feature map obtained to a decoder module comprises: extracting features of the two-dimensional image based on a first encoding unit, and outputting a feature map obtained by the first encoding unit to a second encoding unit and the decoder module; extracting features of the feature map output by the first encoding unit based on the second encoding unit, and outputting a feature map obtained by the second encoding unit to the decoder module, wherein the first encoding unit and the second encoding unit have different structures.

5. The semantic segmentation method of claim 4, wherein, At least one of the first encoding unit and the second encoding unit comprises a plurality of convolution layers, at least one of the plurality of convolution layers uses a long strip-shaped convolution kernel.

6. The semantic segmentation method of claim 5, wherein, The first encoding unit comprises first to third convolution layers arranged in sequence from an input side to an output side, wherein the first and third convolution layers use square convolution kernels, and the second convolution layer uses a long strip-shaped convolution kernel.

7. The semantic segmentation method of claim 3, wherein, The decoder module comprises a plurality of decoding units arranged in sequence from an input side to an output side, and a semantic label classification layer, the decoding unit comprising an up-sampling layer and a plurality of convolution layers.

8. The semantic segmentation method of claim 1, wherein, The second neural network model comprises a plurality of convolution layers and a semantic label classification layer, wherein the convolution layers adopt a 1*1 convolution kernel.

9. An environment perception method, comprising: obtaining point cloud data collected by an unmanned vehicle; determining semantic label information of the point cloud data according to the semantic segmentation method of any one of claims 1-8; determining environment information in which the unmanned vehicle is located according to the semantic label information of the point cloud data.

10. A semantic segmentation apparatus, comprising: a first generation module configured to generate a two-dimensional image according to point cloud data; a first segmentation module configured to perform semantic segmentation processing on the two-dimensional image based on a first neural network model to obtain a semantic segmentation result corresponding to the two-dimensional image, the semantic segmentation result corresponding to the two-dimensional image comprising semantic label information corresponding to each point in the two-dimensional image and feature representation corresponding to each point in the two-dimensional image; a second generation module configured to generate a point cloud feature map according to the semantic segmentation result of the two-dimensional image and the point cloud data, comprising: determining point cloud points in the point cloud data that match each point in the two-dimensional image; splicing the semantic label information corresponding to each point in the two-dimensional image, the feature representation corresponding to each point in the two-dimensional image, and coordinates of the matching point cloud points to obtain spliced feature representation corresponding to each point in the two-dimensional image; and constructing the point cloud feature map according to the spliced feature representation corresponding to each point in the two-dimensional image; a second segmentation module configured to perform semantic segmentation processing on the point cloud feature map based on a second neural network model to obtain semantic label information of the point cloud data.

11. The semantic segmentation device of claim 10, wherein, The two-dimensional image is a depth map, and the first generation module is configured to: determine two-dimensional conversion coordinates of the point cloud data in a spherical coordinate system, wherein the two-dimensional conversion coordinates comprise a yaw angle and a pitch angle; distribute the point cloud data into a plurality of grids according to the two-dimensional conversion coordinates; determine features of each grid according to point cloud points in each grid; construct the depth map according to the features of all grids.

12. A semantic segmentation apparatus, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the semantic segmentation method of any one of claims 1-8 based on instructions stored in the memory.

13. An environment perception device, comprising: an acquisition module configured to obtain point cloud data collected by an unmanned vehicle; the semantic segmentation apparatus of any one of claims 10-12; a determination module configured to determine environment information in which the unmanned vehicle is located according to the semantic label information of the point cloud data.

14. An environment perception device, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the environment perception method of claim 9 based on instructions stored in the memory.

15. A computer-readable storage medium having computer program instructions stored thereon, the instructions being executed by a processor to implement the semantic segmentation method of any one of claims 1-8 or the environment perception method of claim 9.

16. An unmanned vehicle comprising: The semantic segmentation apparatus of any one of claims 10 to 12, or the environment perception device of any one of claims 13-14.

Images