Obstacle recognition method and autonomous vehicle

Abstract

This application discloses an obstacle recognition method and an autonomous vehicle, belonging to the field of autonomous driving technology. The method includes: acquiring point cloud data collected by the autonomous vehicle, the point cloud data indicating obstacles around the autonomous vehicle, the point cloud data including multiple pillars, each pillar including features of at least one data point, and each pillar having the same size; based on each pillar, encoding the features of at least one data point included in the pillar using maximum response coding and attention coding to obtain a second coded feature of the point cloud data, the second coded feature including the first coded feature of each pillar; and performing obstacle recognition on the second coded feature to obtain obstacle information. The solution provided in this application comprehensively considers both the detailed and overall features of the obstacle when recognizing it, improving the accuracy of obstacle recognition and thus improving the safety of autonomous vehicle operation.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an obstacle recognition method and an autonomous vehicle. Background Technology

[0002] With the rapid development of autonomous driving technology, autonomous vehicles can identify their surroundings and obstacles, enabling them to plan routes based on these obstacles and ensure that they avoid them, thus improving the safety of autonomous driving. However, the current methods for identifying obstacles are not very accurate. Summary of the Invention

[0003] This application provides an obstacle recognition method and an autonomous vehicle. The method comprehensively considers both the detailed and overall features of obstacles, improving the accuracy of obstacle recognition and thus enhancing the safety of the autonomous vehicle. The technical solution is as follows:

[0004] On the one hand, an obstacle recognition method is provided, the method comprising:

[0005] Acquire point cloud data collected by an autonomous vehicle, the point cloud data indicating obstacles around the autonomous vehicle, the point cloud data including multiple pillars, each pillar including features of at least one data point, and each pillar having the same size;

[0006] Based on each of the pillars, the features of at least one data point included in the pillar are encoded using maximum response coding and attention coding to obtain a second encoded feature of the point cloud data, wherein the second encoded feature includes the first encoded feature of each of the pillars;

[0007] Obstacle identification is performed on the second encoded feature to obtain obstacle information of the obstacle.

[0008] On one hand, an obstacle recognition device is provided, the device comprising:

[0009] An acquisition module is used to acquire point cloud data collected by an autonomous vehicle. The point cloud data indicates obstacles around the autonomous vehicle. The point cloud data includes multiple pillars, each pillar includes features of at least one data point, and each pillar has the same size.

[0010] An encoding module is used to encode the features of at least one data point included in each of the pillars using maximum response encoding and attention encoding to obtain a second encoded feature of the point cloud data, wherein the second encoded feature includes the first encoded feature of each of the pillars.

[0011] The identification module is used to identify obstacles based on the second encoded features to obtain obstacle information of the obstacles.

[0012] In one possible implementation, the encoding module includes:

[0013] An extraction unit is used to extract features from at least one data point included in each pillar to obtain features of a preset dimension and corresponding weights for each data point.

[0014] The extraction unit is further configured to extract the maximum feature corresponding to each dimension from the features of at least one data point included in the pillar;

[0015] An encoding unit is used to perform attention encoding on the features and corresponding weights of the preset dimension to obtain attention features;

[0016] The summation unit is used to sum the maximum feature corresponding to each dimension and the feature corresponding to each dimension in the attention feature to obtain the second encoded feature.

[0017] In one possible implementation, the extraction unit is used for:

[0018] For each pillar, the features of at least one data point of the pillar are subjected to dimensionality-upgrading processing to obtain dimensionality-upgraded features;

[0019] Feature extraction is performed on the upgraded features to obtain the features and corresponding weights of the at least one data point in a preset dimension.

[0020] In one possible implementation, the extraction unit is used for:

[0021] High-dimensional feature extraction is performed on the upgraded feature to obtain a feature of a preset dimension for the at least one data point, wherein the preset dimension is greater than the dimension of the upgraded feature;

[0022] The features of the at least one data point in a preset dimension are normalized to obtain the weights corresponding to the features of the at least one data point in the preset dimension.

[0023] In one possible implementation, the encoding unit is used for:

[0024] The features of the preset dimension and their corresponding weights are weighted to obtain weighted features;

[0025] The attention feature is obtained by summing the features of at least one data point included in the weighted feature.

[0026] In one possible implementation, the identification module includes:

[0027] A dimensionality reduction unit is used to perform dimensionality reduction processing on the second encoded feature to obtain a dimensionality-reduced feature;

[0028] The identification unit is used to identify obstacles from the reduced-dimensional features to obtain obstacle information.

[0029] In one possible implementation, the dimensionality reduction unit is used for:

[0030] The second encoded features are processed by convolutional networks using a first convolutional network and a second convolutional network respectively, resulting in processed first and second convolutional features.

[0031] The first and second convolutional features are concatenated to obtain concatenated features.

[0032] The cascaded features are convolutionally processed to obtain the dimensionality-reduced features.

[0033] In one possible implementation, the first convolutional network includes multiple convolutional layers, and the dimensionality reduction unit is used for:

[0034] The second encoded feature is convolved using each of the multiple convolutional layers to obtain multiple sub-convolutional features;

[0035] The multiple sub-convolutional features and the second encoding features are normalized to obtain multiple normalized features;

[0036] The normalized features are summed to obtain the first convolutional feature.

[0037] In one possible implementation, the dimensionality reduction features include multiple layers of sub-features, each layer having a different dimension, and the recognition unit is used for:

[0038] The last sub-feature included in the dimensionality reduction feature is deconvolved to obtain a deconvolution feature with the same dimension as the second-to-last sub-feature.

[0039] The deconvolutional features and the last layer sub-features are concatenated to obtain the concatenated features;

[0040] Obstacle identification is performed on the splicing features to obtain obstacle information of the obstacles.

[0041] In one possible implementation, the obstacle recognition method is performed by a target detection model, and the apparatus further includes:

[0042] The acquisition module is also used to acquire sample point cloud data and sample obstacle information. The sample point cloud data indicates sample obstacles around the autonomous vehicle. The sample point cloud data includes multiple sample pillars, each sample pillar includes features of at least one data point, and each sample pillar has the same size.

[0043] The recognition module is also used to call the target detection model to perform obstacle recognition on each of the sample pillars in the sample point cloud data to obtain predicted obstacle information;

[0044] The device further includes a training module, which is used to train the target detection model based on the sample obstacle information and the predicted obstacle information to obtain the trained target detection model.

[0045] On one hand, an autonomous vehicle is provided, the autonomous vehicle including one or more processors and one or more memories, the one or more memories storing at least one piece of program code, the at least one piece of program code being loaded and executed by the one or more processors to perform the operations performed by the obstacle recognition method as described in any of the above possible implementations.

[0046] On one hand, a computer-readable storage medium is provided, which stores at least one piece of program code, which is loaded and executed by a processor to perform the operations performed by the obstacle recognition method as described in any of the above possible implementations.

[0047] On one hand, a computer program or computer program product is provided, the computer program or computer program product comprising: computer program code, which, when executed by an autonomous vehicle, causes the autonomous vehicle to perform the operations performed by the obstacle recognition method as described in any of the above possible implementations.

[0048] This application provides a scheme for obstacle identification. After acquiring point cloud data including obstacles, the autonomous vehicle encodes the features of data points of each pillar in the point cloud data through maximum response coding and attention coding. The maximum response coding can extract the maximum value of the features to obtain the local features of the obstacle, which are used to describe the detailed features of the obstacle. The attention coding can obtain the global features of the obstacle by weighting the features, which are used to describe the overall features of the obstacle. Therefore, obstacle identification comprehensively considers the detailed features and overall features of the obstacle, improves the accuracy of obstacle identification, and thus improves the safety of autonomous vehicle driving. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart of an obstacle recognition method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart of an obstacle recognition method provided in an embodiment of this application;

[0053] Figure 4 This is a flowchart of feature encoding provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a convolutional network provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of another convolutional network structure provided in an embodiment of this application;

[0056] Figure 7 This is a flowchart of another obstacle recognition method provided in an embodiment of this application;

[0057] Figure 8 This is a schematic diagram illustrating the comparison of obstacle recognition results provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of an obstacle recognition device provided in an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of another obstacle recognition device provided in an embodiment of this application;

[0060] Figure 11 This is a schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application;

[0061] Figure 12 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0063] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first coding feature may be referred to as a second coding feature, and a second coding feature may be referred to as a first coding feature.

[0064] As used in this application, the terms "at least one", "multiple", "each", and "any" are used in the following ways: at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any refers to any one of the multiple. For example, multiple pillars include three pillars, each refers to each of the three pillars, and any refers to any one of the three pillars, which can be the first, the second, or the third.

[0065] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the point cloud data involved in this application was obtained with full authorization. Furthermore, the aforementioned information and data, after processing, are used in big data application scenarios and cannot be identified to any natural person or establish a specific association with them.

[0066] In some embodiments, the obstacle recognition method provided in this application is performed by an autonomous vehicle. The autonomous vehicle can be any device capable of autonomous driving, such as a driverless car, intelligent robot, etc. In some embodiments, the autonomous vehicle includes vehicles traveling on the ground (e.g., cars, trucks, buses, etc.), vehicles traveling in the air (e.g., drones, airplanes, helicopters, etc.), and vehicles traveling on water or in water (e.g., ships, submarines, etc.). The autonomous vehicle may or may not accommodate one or more passengers. Furthermore, the autonomous vehicle can be applied to unmanned delivery fields, such as express delivery and food delivery.

[0067] In other embodiments, the obstacle recognition method provided in this application is executed by an autonomous vehicle and a server. The server can be a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0068] It should be noted that the embodiments of this application do not limit the subject that performs the obstacle recognition method.

[0069] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application, such as... Figure 1 As shown, the implementation environment includes an autonomous vehicle 101 and a server 102, which are connected via a wireless or wired network.

[0070] Server 102 provides autonomous driving-related services to autonomous vehicle 101. In some embodiments, autonomous vehicle 101 collects point cloud data and sends the collected point cloud data to server 102. Server 102 encodes the features of at least one data point of each pillar in the point cloud data using maximum response coding and attention coding to obtain a second coded feature of the point cloud data. The second coded feature includes the first coded feature of each pillar. Obstacle identification is performed on the second coded feature to obtain obstacle information. The identified obstacle information is sent to autonomous vehicle 101, which can then plan a route based on the received obstacle information.

[0071] Figure 2 This is a flowchart illustrating an obstacle recognition method provided in an embodiment of this application. This embodiment uses an autonomous vehicle as an example for illustrative purposes, and includes:

[0072] 201. The autonomous vehicle acquires point cloud data collected by the autonomous vehicle. The point cloud data indicates obstacles around the autonomous vehicle. The point cloud data includes multiple pillars, each pillar includes features of at least one data point, and each pillar is the same size.

[0073] Point cloud data refers to a set of data points, each with features of different dimensions. In some embodiments, the point cloud data is three-dimensional, meaning each data point in the point cloud data has features of three dimensions. An obstacle refers to an object in the environment surrounding the autonomous vehicle. In some embodiments, the obstacle can be a car, bus, tree, bush, lamppost, or other type; this application embodiment does not limit this. In some embodiments, the obstacle may or may not affect the autonomous vehicle's driving route; this application embodiment does not limit this. Pillars are pre-set according to the coordinate system of the point cloud data. Based on the position of each data point in the point cloud data collected by the autonomous vehicle, each data point in the point cloud data can be assigned to a corresponding pillar, and each pillar includes the features of at least one data point. In this application embodiment, when encoding the point cloud data, the data points within each pillar are processed on a pillar-by-pillar basis, and subsequently, encoded features are obtained.

[0074] In some embodiments, the coordinate system of the point cloud data is divided according to a preset size to obtain multiple pillars, and the data points included in the point cloud data can fall within the corresponding pillars. Therefore, each pillar can include the features of at least one data point.

[0075] In this embodiment, the autonomous vehicle collects point cloud data of obstacles around it. Subsequently, the collected point cloud data can be used to identify obstacles, and then a route can be planned based on the identified obstacle information to prevent the autonomous vehicle from colliding with the obstacles.

[0076] 202. For each pillar, the autonomous vehicle uses maximum response coding and attention coding to encode the features of at least one data point included in the pillar to obtain the second coded features of the point cloud data. The second coded features include the first coded features of each pillar.

[0077] The maximum response encoding is used to extract local key features for each pillar, which refers to the features of some data points within the pillar. For example, the local key feature is the feature of the data point with the largest feature within the pillar. The attention encoding is used to extract global key features for each pillar, which refers to the fused features obtained by fusing the features of all data points within the pillar.

[0078] In this embodiment of the application, the autonomous vehicle encodes the features of the data points included in each pillar using maximum response coding and attention coding respectively, thereby obtaining the local key features and global key features of each pillar, and then obtaining the second coding feature of the point cloud data. The second coding feature includes the first coding feature of each pillar among multiple pillars.

[0079] 203. The autonomous vehicle performs obstacle recognition on the second coded features to obtain obstacle information.

[0080] The obstacle information is used to indicate obstacles. In some embodiments, the obstacle information includes at least one of the following: obstacle type, size, distance from the autonomous vehicle, center point of the obstacle, and yaw angle of the obstacle; or the obstacle information may also include other types of information, which are not limited in this application embodiment.

[0081] After obtaining the second encoded feature of the point cloud data collected by the autonomous vehicle, obstacle information of the obstacle in the point cloud data can be obtained based on the second encoded feature.

[0082] This application provides a scheme for identifying obstacles. After an autonomous vehicle acquires point cloud data including obstacles, it encodes the features of data points of each pillar in the point cloud data through maximum response coding and attention coding. The maximum response coding can extract the maximum value of the features to obtain the local features of the obstacle, which are used to describe the detailed features of the obstacle. The attention coding can obtain the global features of the obstacle by weighting the features, which are used to describe the overall features of the obstacle. Therefore, when identifying obstacles, both the detailed features and the overall features of the obstacle are considered, which improves the accuracy of obstacle identification and thus improves the safety of autonomous vehicle driving.

[0083] Figure 3 This is a flowchart illustrating an obstacle recognition method provided in an embodiment of this application. This embodiment uses an autonomous vehicle as an example for illustrative purposes, and includes:

[0084] 301. The autonomous vehicle acquires point cloud data collected by the autonomous vehicle. The point cloud data indicates obstacles around the autonomous vehicle. The point cloud data includes multiple pillars, each pillar includes features of at least one data point, and each pillar is the same size.

[0085] In some embodiments, the autonomous vehicle is equipped with radar to detect surrounding obstacles and collect point cloud data. Optionally, the radar includes lidar, millimeter-wave radar, ultrasonic radar, or other radars, which are not limited in the embodiments of this application.

[0086] In other embodiments, the autonomous vehicle is equipped with a camera to capture images of surrounding obstacles, thereby collecting point cloud data. Optionally, the camera can be a depth camera, a binocular camera, or other types of cameras; this application does not limit the specific type of camera.

[0087] For point cloud data, the data points include features in multiple dimensions. In some embodiments, the data points include left-right space, front-back space, top-bottom space, reflection intensity, and timestamp information. Optionally, the left-right space, front-back space, and top-bottom space can be represented by x, y, and z, respectively.

[0088] 302. For each pillar, the autonomous vehicle extracts features from at least one data point included in the pillar to obtain the features of each data point in a preset dimension and the corresponding weight.

[0089] Here, the preset dimension feature means that the dimension of the feature of each data point is a preset dimension. In some embodiments, for each data point in each pillar, feature extraction can be performed on each data point in each pillar to obtain the preset dimension feature of each data point, and the weight corresponding to the preset dimension feature of each data point can also be obtained.

[0090] In this embodiment of the application, after the autonomous vehicle acquires point cloud data, it can divide the multiple data points of the point cloud data into multiple pillars, extract features from the point cloud data in the manner of each pillar, extract the coded features of the point cloud data according to each pillar, and then perform obstacle recognition based on the obtained coded features.

[0091] In some embodiments, for each pillar, the features of at least one data point of the pillar are subjected to dimensionality upscaling to obtain dimensionality upscaling features, and feature extraction is performed on the dimensionality upscaling features to obtain features of at least one data point with a preset dimension and corresponding weights.

[0092] In this context, dimensionality enhancement refers to increasing the dimension of the features of data points to obtain dimensionality-enhanced features. Optionally, in this embodiment, the dimensionality-enhanced features are obtained based on the original features of at least one data point of each pillar.

[0093] In some embodiments, the dimensions of at least one data point in the point cloud data include x-absolute coordinates, y-absolute coordinates, z-absolute coordinates, reflection intensity, and timestamp information. Based on the characteristics of at least one data point in each dimension, the differences between the x-absolute coordinates, y-absolute coordinates, and z-absolute coordinates and the center and minimum coordinates of the corresponding pillar are calculated respectively. The differences of each data point in each dimension can be used as a new dimension, thus realizing the dimensionality upgrade of each data point and obtaining the dimensionality upgrade features of each data point.

[0094] For example, the center coordinates of each pillar and the minimum coordinates of the point cloud data are obtained. The differences between the x-coordinate, y-coordinate, and z-coordinate and the center coordinates, as well as the differences between the z-coordinate and the minimum coordinates, are then obtained. These x-coordinate, y-coordinate, z-coordinate, reflection intensity, timestamp information, and the differences obtained based on each coordinate are used to determine 11 dimensions of features, achieving dimensionality upscaling for at least one data point. Specifically, the differences between the x-coordinate, y-coordinate, and z-coordinate and the center coordinates include the differences between the x-coordinate and the x-coordinate within the center coordinates, the y-coordinate and the y-coordinate within the center coordinates, and the z-coordinate and the z-coordinate within the center coordinates. The differences between the x-coordinate, y-coordinate, and z-coordinate and the minimum coordinates are also calculated.

[0095] Optionally, high-dimensional feature extraction is performed on the upgraded features to obtain features of at least one data point with a preset dimension, where the preset dimension is greater than the dimension of the upgraded features. The features of the preset dimension of the at least one data point are then normalized to obtain the weights corresponding to the features of the preset dimension of the at least one data point.

[0096] In some embodiments, the autonomous vehicle invokes a first neural network to perform high-dimensional feature processing on the increased-dimensional features to obtain features of a preset dimension. Optionally, the first neural network needs to be pre-trained to enable it to perform high-dimensional feature extraction to obtain features of the preset dimension.

[0097] For example, the sample's upgraded features and the sample's high-dimensional features of a preset dimension are obtained. The first neural network is used to process the sample's upgraded features to obtain the predicted high-dimensional features of the preset dimension. The first neural network is then trained based on the difference between the sample's high-dimensional features and the predicted high-dimensional features, thus obtaining the trained first neural network.

[0098] In other embodiments, the process of normalizing features of at least one data point in a preset dimension can also be implemented using a second neural network. The autonomous vehicle invokes the second neural network to normalize the features in the preset dimension, thereby obtaining the weights corresponding to the features in the preset dimension. Optionally, the second neural network needs to be pre-trained to enable it to perform normalization processing.

[0099] For example, obtain sample features of a preset dimension and sample weights corresponding to the sample features of the preset dimension, use a second neural network to normalize the sample features of the preset dimension to obtain the prediction weights corresponding to the sample features of the preset dimension, and train the second neural network based on the difference between the sample weights corresponding to the sample features of the preset dimension and the prediction weights to obtain the trained second neural network.

[0100] It should be noted that the first neural network and the second neural network in the embodiments of this application are two networks in one model. The first neural network and the second neural network are trained simultaneously to obtain the trained neural network.

[0101] 303. Autonomous vehicles extract the maximum feature corresponding to each dimension from the features of at least one data point included in the pillar.

[0102] In this context, for each of the multiple pillars, at least one data point in each pillar has features of a preset dimension. The autonomous vehicle can extract the maximum feature in each dimension from the features of at least one data point in that pillar.

[0103] 304. Autonomous vehicles perform attention encoding on features and corresponding weights of preset dimensions to obtain attention features.

[0104] In this embodiment, after obtaining the features and corresponding weights of the preset dimension, the features of the preset dimension can represent the features of each pillar in the point cloud data. By performing attention encoding on the features and corresponding weights of the preset dimension, the obtained attention features comprehensively consider the features of each data point in the point cloud data, so as to consider the global features in the point cloud data when performing obstacle recognition.

[0105] In some embodiments, the features of a preset dimension and their corresponding weights are weighted to obtain weighted features, and the features of at least one data point included in the weighted features are summed to obtain attention features.

[0106] It should be noted that the weighting process and the acquisition of attention features in this embodiment are both performed on a pillar basis. Therefore, the features of at least one data point in a pillar and the corresponding weights are weighted to obtain the weighted features of the pillar. Then, the features of at least one data point in each dimension of the weighted features of the pillar are summed to obtain the features in each dimension. In this way, the features in multiple dimensions constitute the attention features of the pillar.

[0107] It should be noted that steps 303 and 304 can be executed simultaneously, or steps 303 can be executed first and then steps 304, or steps 304 can be executed first and then steps 303. This application embodiment does not limit this.

[0108] 305. The autonomous vehicle sums the maximum feature and the attention feature corresponding to each dimension to obtain the second encoded feature.

[0109] In this embodiment of the application, the autonomous vehicle obtains the maximum feature and attention feature corresponding to each pillar in each dimension, with each pillar as the unit. The maximum feature refers to the local key feature of the corresponding pillar in the corresponding dimension, and the attention feature refers to the global key feature of the corresponding pillar in the corresponding dimension. Therefore, the feature values ​​corresponding to each dimension in the maximum feature and attention feature corresponding to each dimension are summed to obtain the second encoded feature. The obtained second encoded feature contains both the local key features and the global key features of the point cloud data.

[0110] In some embodiments, the maximum feature refers to the feature with the largest value among the features corresponding to each dimension. For example, if the features in any dimension include 25, 30, 33, and 45, then the maximum feature in that dimension is 45.

[0111] It should be noted that in the embodiments of this application, steps 302-305 are performed by the MAPE pillar coding component, thereby obtaining the second coding feature.

[0112] The following examples illustrate how this application obtains the second coding feature. (See also...) Figure 4 The process for obtaining the second encoded feature is as follows:

[0113] For each pillar, each data point includes 5-dimensional features. By performing dimensionality upscaling on the features of the data points of each pillar, 11-dimensional features are obtained. Then, high-dimensional feature extraction is performed on the 11-dimensional features to obtain D-dimensional features and their corresponding weights. From the features of at least one data point included in the pillar, the maximum feature corresponding to each dimension is extracted to obtain 1*D features. Attention encoding is then performed on the D-dimensional features and their corresponding weights to obtain 1*D features. Finally, the two 1*D features are summed to obtain the second 1*D encoded features. This second 1*D encoded feature is the pillar shown in the final diagram.

[0114] It should be noted that steps 302-305 in this embodiment are one possible implementation of step 202 above. Alternatively, steps 302-305 can be omitted, and step 202 can be executed directly.

[0115] 306. The autonomous vehicle performs dimensionality reduction processing on the second encoded feature to obtain the dimensionality-reduced feature.

[0116] Dimensionality reduction refers to reducing the dimensionality of the second encoded feature on a horizontal scale. In some embodiments, the second encoded feature is a three-dimensional feature, and dimensionality reduction refers to reducing the dimensionality of the second encoded feature from a two-dimensional perspective to obtain a dimensionality-reduced feature. For example, the second encoded feature includes features with three dimensions, namely H, W, and D. Dimensionality reduction of the second encoded feature refers to reducing the dimensionality of the second encoded feature on the horizontal dimension formed by the H*W dimensions to obtain a dimensionality-reduced feature.

[0117] In some embodiments, a first convolutional network and a second convolutional network are used to perform convolution processing on the second encoded features to obtain processed first convolutional features and second convolutional features. The first convolutional features and the second convolutional features are concatenated to obtain concatenated features. The concatenated features are then subjected to convolution processing to obtain dimensionality-reduced features.

[0118] In this embodiment, the first convolutional network and the second convolutional network are different convolutional networks. By performing convolutional processing on the second encoded features separately, the dimensionality reduction of the second encoded features can be achieved. Furthermore, by using two convolutional networks, the number of parameters of the first convolutional network is reduced while maintaining the feature representation capability, thereby reducing the network capacity.

[0119] Optionally, the first convolutional network actually has multiple repeating structures, which can also be understood as the first convolutional network being executed multiple times in order to complete the convolutional processing of the second encoded features.

[0120] For example, the first convolutional network is Figure 5 The RepConv (a type of convolutional network) shown has a second convolutional network that is an intermediate 1*1 convolutional network. The second encoded feature is first processed by the 1*1 convolutional network, and then processed by RepConv and the 1*1 convolutional network respectively to obtain the first convolutional feature and the second convolutional feature. The first convolutional feature and the second convolutional feature are then concatenated to obtain the concatenated feature. The concatenated feature is then convolved to obtain the dimensionality-reduced feature.

[0121] Optionally, the second encoded feature is convolved by each of the multiple convolutional layers to obtain multiple sub-convolutional features. The multiple sub-convolutional features and the second encoded feature are normalized to obtain multiple normalized features. The normalized features are then summed to obtain the first convolutional feature.

[0122] In some embodiments, the first convolutional network is a reparameterized network, meaning that during training, the first convolutional network uses each of the multiple convolutional layers to convolve the second encoded features to obtain multiple sub-convolutional features. The multiple sub-convolutional features and the second encoded features are then normalized to obtain multiple normalized features. The normalized features are then summed to obtain the first convolutional feature. During deployment, the multiple convolutional layers are merged to obtain a single convolutional layer, which is then used to convolve the second encoded features.

[0123] For example, such as Figure 6 The diagram illustrates the specific structures of the first and second convolutional networks of this application. The 'd' part represents the overall structure of the first and second convolutional networks, as shown above. Figure 5 As shown, the first convolutional network further consists of RepConv and Leaky-ReLU (an activation function) in part c. The specific structure of RepConv in part c is shown in part b, which includes a 1x1 convolutional layer, a first normalization layer (BN1) connected to the 1x1 convolutional layer, a 3x3 convolutional layer, a second normalization layer (BN2) connected to the 3x3 convolutional layer, and a third normalization layer (BN3). Furthermore, part b refers to the network structure during training; during deployment, the deployed structure is merged into the structure shown in part a, which consists of two layers: a 3x3 convolutional layer and an activation layer.

[0124] 307. Autonomous vehicles use reduced-dimensional features to identify obstacles and obtain obstacle information.

[0125] The dimensionality reduction feature indicates an obstacle. Therefore, after the autonomous vehicle obtains the dimensionality reduction feature, it can identify the obstacle's category, size, distance from the autonomous vehicle, center point of the obstacle, and yaw angle, or it can also obtain other obstacle information. This application embodiment does not limit this.

[0126] In some embodiments, the obstacle information includes class HM (object detection probability), Iou (Intersection over Union), Loc (position deviation), Dimen (size), and orien (yaw angle). Here, class HM indicates whether there is an object and what the probability of it is; Iou indicates the degree of overlap between the object and the object in the actual scene; Loc indicates the deviation between the object's position and the object's position in the actual scene; Dimen indicates the size information of the object; and orien indicates the object's orientation.

[0127] In some embodiments, the dimensionality reduction feature includes multiple layers of sub-features, each with a different dimension. Obstacle recognition is performed on the dimensionality reduction feature to obtain obstacle information of the obstacle, including: performing deconvolution on the last layer of sub-features included in the dimensionality reduction feature to obtain a deconvolution feature with the same dimension as the penultimate layer of sub-features; concatenating the deconvolution feature and the last layer of sub-features to obtain a concatenated feature; and performing obstacle recognition on the concatenated feature to obtain obstacle information of the obstacle.

[0128] For example, see Figure 7 This illustrates the principle of obstacle recognition in this application; see [link / reference]. Figure 7 After an autonomous vehicle inputs point cloud data, a recognition network identifies the point cloud data to obtain the network's output. This output includes five parameters: the presence of an object, the degree of overlap between the object and objects in the actual scene, the deviation between the object's position and that of objects in the actual scene, the object's size, and the object's orientation. The final recognition result is then determined based on these five output parameters. Specifically, after acquiring the point cloud data, the recognition network encodes the data to obtain encoded features. These encoded features are then convolved to obtain the five sub-feature layers shown in Figure CRV Net. The last sub-feature layer is deconvolved to obtain features of the same size as the second-to-last layer. These features are then concatenated with the second-to-last layer's sub-features and convolved to obtain the output result.

[0129] It should be noted that steps 306-307 in this embodiment are one possible implementation of step 203 above. In this embodiment, step 203 can also be executed directly.

[0130] The following provides examples illustrating the detection results of this application. In some embodiments, a comparison is made between actual annotation data, annotation data obtained through related technologies, and annotation data obtained through the method provided in this application. See also... Figure 8 The first column contains three types of actual labeled data. The second column contains labeled data obtained by detecting actual point cloud data using related technologies. Data within the boxes is data not identified by related technologies. The third column contains labeled data obtained by the solution provided in this application, which is the same as the actual labeled data in the first column. Therefore, it also indicates the accuracy of obstacle recognition in this application.

[0131] It should be noted that the method in this application embodiment can also be executed by an object detection model. The training process of the object detection model includes: acquiring sample point cloud data and sample obstacle information. The sample point cloud data indicates sample obstacles around the autonomous vehicle. The sample point cloud data includes multiple sample pillars. Each sample pillar includes features of at least one data point, and each sample pillar has the same size. The object detection model is invoked to identify obstacles for each sample pillar in the sample point cloud data to obtain predicted obstacle information. The object detection model is trained based on the sample obstacle information and the predicted obstacle information to obtain the trained object detection model.

[0132] In some embodiments, the object detection model is a StrongPillars (a detection model structure) model, where Methed is the network format, stages is the number of network training iterations, View is the point cloud processing angle, NDS is an indicator of network performance, mAP (mean Average Precision) is the average accuracy across all classes, car (car), Truck (truck), Bus (bus), Trailer (trailer), CV (construction vehicle), Ped (pedestrian), Motor (motorcycle), BC (bicycle), TC (traffic cone), and Barrier (crossable obstacle).

[0133] See Table 1 for the data details for each network.

[0134] Table 1

[0135]

[0136]

[0137]

[0138] Furthermore, this application also proposes advantages in detection speed for different networks. See Table 2, which shows the speed for different networks, where detection speed is expressed in frame rate:

[0139] Table 2

[0140] Methods SPConv Frame rate NDS (%) mAP(%) Pointpillars (0.2m) no 31 60.22 50.26 Point Pillars (0.075m) no 9 59.51 48.63 Point Pillars (0.075m) yes 14 67.21 59.67 PllarNet-34 (0.075m) yes 12 67.55 60.16 StrongPillars-s(0.15m) no 24 67.22 60.00 StrongPillars-m(0.15m) no 16 68.19 61.72

[0141] The obstacle recognition method provided in this application involves an autonomous vehicle acquiring point cloud data including obstacles. Then, through maximum response coding and attention coding, the data point features of each pillar in the point cloud data are encoded. Maximum response coding extracts the maximum value of the features to obtain the local features of the obstacle, which describe the detailed features of the obstacle. Attention coding weights the features to obtain the global features of the obstacle, which describe the overall features of the obstacle. By comprehensively considering both the detailed and overall features of the obstacle during obstacle recognition, the accuracy of obstacle recognition is improved, thereby enhancing the safety of the autonomous vehicle.

[0142] Furthermore, by using dimensionality reduction, autonomous vehicles can simplify the data consumption for obstacle recognition and improve recognition speed. Additionally, it avoids the problem of excessively large network structures causing autonomous vehicles to malfunction during deployment, achieving a quantized deployment effect on autonomous vehicles and thus ensuring their ability to correctly recognize obstacles.

[0143] Figure 9 This is a schematic diagram of the structure of an obstacle recognition device provided in an embodiment of this application. See also... Figure 9 The device includes:

[0144] The acquisition module 901 is used to acquire point cloud data collected by the autonomous vehicle. The point cloud data indicates obstacles around the autonomous vehicle. The point cloud data includes multiple pillars, each pillar includes features of at least one data point, and each pillar has the same size.

[0145] Encoding module 902 is used to encode the features of at least one data point included in each of the pillars using maximum response encoding and attention encoding to obtain a second encoded feature of the point cloud data, wherein the second encoded feature includes the first encoded feature of each of the pillars;

[0146] The identification module 903 is used to identify obstacles based on the second encoded feature to obtain obstacle information of the obstacle.

[0147] In one possible implementation, the encoding module 902 includes:

[0148] The extraction unit 9021 is used to extract features from at least one data point included in each of the pillars to obtain features of a preset dimension and corresponding weights for each data point.

[0149] The extraction unit 9021 is further configured to extract the maximum feature corresponding to each dimension from the features of at least one data point included in the pillar;

[0150] Encoding unit 9022 is used to perform attention encoding on the features and corresponding weights of the preset dimension to obtain attention features;

[0151] The summation unit 9023 is used to sum the maximum feature corresponding to each dimension and the feature corresponding to each dimension in the attention feature to obtain the second encoded feature.

[0152] In one possible implementation, the extraction unit 9021 is used for:

[0153] For each pillar, the features of at least one data point of the pillar are subjected to dimensionality-upgrading processing to obtain dimensionality-upgraded features;

[0154] Feature extraction is performed on the upgraded features to obtain the features and corresponding weights of the at least one data point in a preset dimension.

[0155] In one possible implementation, the extraction unit 9021 is used for:

[0156] High-dimensional feature extraction is performed on the upgraded feature to obtain a feature of a preset dimension for the at least one data point, wherein the preset dimension is greater than the dimension of the upgraded feature;

[0157] The features of the at least one data point in a preset dimension are normalized to obtain the weights corresponding to the features of the at least one data point in the preset dimension.

[0158] In one possible implementation, the encoding unit 9022 is used for:

[0159] The features of the preset dimension and their corresponding weights are weighted to obtain weighted features;

[0160] The attention feature is obtained by summing the features of at least one data point included in the weighted feature.

[0161] In one possible implementation, the identification module 903 includes:

[0162] Dimensionality reduction unit 9031 is used to perform dimensionality reduction processing on the second encoded feature to obtain dimensionality reduction feature;

[0163] The identification unit 9032 is used to identify obstacles from the reduced-dimensional features to obtain obstacle information.

[0164] In one possible implementation, the dimensionality reduction unit 9031 is used for:

[0165] The second encoded features are processed by convolutional networks using a first convolutional network and a second convolutional network respectively, resulting in processed first and second convolutional features.

[0166] The first and second convolutional features are concatenated to obtain concatenated features.

[0167] The cascaded features are convolutionally processed to obtain the dimensionality-reduced features.

[0168] In one possible implementation, the first convolutional network includes multiple convolutional layers, and the dimensionality reduction unit 9031 is used for:

[0169] The second encoded feature is convolved using each of the multiple convolutional layers to obtain multiple sub-convolutional features;

[0170] The multiple sub-convolutional features and the second encoding features are normalized to obtain multiple normalized features;

[0171] The normalized features are summed to obtain the first convolutional feature.

[0172] In one possible implementation, the dimensionality reduction feature includes multiple layers of sub-features, each with a different dimension, and the recognition unit 9032 is used for:

[0173] The last sub-feature included in the dimensionality reduction feature is deconvolved to obtain a deconvolution feature with the same dimension as the second-to-last sub-feature.

[0174] The deconvolutional features and the last layer sub-features are concatenated to obtain the concatenated features;

[0175] Obstacle identification is performed on the splicing features to obtain obstacle information of the obstacles.

[0176] In one possible implementation, the obstacle recognition method is performed by a target detection model, and the acquisition module 901 is further configured to acquire sample point cloud data and sample obstacle information. The sample point cloud data indicates sample obstacles around the autonomous vehicle. The sample point cloud data includes multiple sample pillars, each sample pillar includes features of at least one data point, and each sample pillar has the same size.

[0177] The recognition module 903 is also used to call the target detection model to perform obstacle recognition on each of the sample pillars in the sample point cloud data to obtain predicted obstacle information;

[0178] The device further includes a training module 904, which is also used to train the target detection model based on the sample obstacle information and the predicted obstacle information to obtain the trained target detection model.

[0179] It should be noted that the obstacle recognition device provided in the above embodiments is only illustrated by the division of the above functional modules when recognizing obstacles. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the autonomous vehicle can be divided into different functional modules to complete all or part of the functions described above. In addition, the obstacle recognition device and the obstacle recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0180] Figure 11 A structural block diagram of an autonomous vehicle 1100 provided in an exemplary embodiment of this application is shown. The autonomous vehicle 1100 includes a processor 1101, which may include one or more computer-readable storage media, which may be non-transitory. In some embodiments, the non-transitory computer-readable storage media in the processor 1101 is used to store at least one line of program code, which is executed by the processor 1101 to implement the operations performed by the autonomous vehicle in the obstacle recognition method provided in the method embodiments of this application.

[0181] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the autonomous vehicle 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0182] Figure 12 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1200 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The memory 1202 stores at least one line of program code, which is loaded and executed by the processor 1201 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0183] The server 1200 is used to execute the steps performed by the server in the above method embodiments.

[0184] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a computer device to perform the obstacle recognition method in the above embodiments. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0185] In an exemplary embodiment, a computer program or computer program product is also provided, which includes computer program code that, when executed by a computer, causes the computer to implement the obstacle recognition method in the above embodiments.

[0186] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0187] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An obstacle recognition method characterized by, The method comprises: acquiring point cloud data collected by an autonomous vehicle, the point cloud data indicating obstacles around the autonomous vehicle, the point cloud data comprising a plurality of pillars, each pillar comprising a feature of at least one data point, and each pillar having the same size; based on each pillar, performing encoding processing on the feature of the at least one data point included in the pillar by using maximum response encoding and attention encoding to obtain a second encoding feature of the point cloud data, the second encoding feature comprising a first encoding feature of each pillar; performing obstacle identification on the second encoding feature to obtain obstacle information of the obstacles.

2. The method of claim 1, wherein, The method comprises: for each pillar, performing feature extraction on the feature of the at least one data point included in the pillar to obtain a feature of a preset dimension of each data point and a corresponding weight; extracting a maximum feature corresponding to each dimension from the feature of the at least one data point; performing attention encoding on the feature of the preset dimension and the corresponding weight to obtain an attention feature; summing the maximum feature corresponding to each dimension and the feature corresponding to each dimension in the attention feature to obtain the second encoding feature.

3. The method of claim 2, wherein, The method comprises: for each pillar, performing dimension lifting processing on the feature of the at least one data point to obtain a lifted feature; performing feature extraction on the lifted feature to obtain a feature of a preset dimension of the at least one data point and a corresponding weight.

4. The method of claim 3, wherein, The method comprises: performing high-dimensional feature extraction on the lifted feature to obtain a feature of a preset dimension of the at least one data point, the preset dimension being greater than a dimension of the lifted feature; performing normalization processing on the feature of the preset dimension of the at least one data point to obtain a weight corresponding to the feature of the preset dimension of the at least one data point.

5. The method of claim 2, wherein, The method comprises: performing weighting processing on the feature of the preset dimension and the corresponding weight to obtain a weighted feature; summing the feature of the at least one data point included in the weighted feature to obtain the attention feature.

6. The method of claim 1, wherein, The method comprises: performing dimension reduction processing on the second encoding feature to obtain a reduced feature; performing obstacle identification on the reduced feature to obtain obstacle information of the obstacles.

7. The method of claim 6, wherein, The method comprises: performing dimension reduction processing on the second encoding feature to obtain a reduced feature; The first convolutional network and the second convolutional network are used to respectively perform convolutional processing on the second encoded feature, to obtain a processed first convolutional feature and a processed second convolutional feature; The first convolutional feature and the second convolutional feature are concatenated to obtain a concatenated feature; The concatenated feature is subjected to convolutional processing to obtain the dimension-reduced feature.

8. The method of claim 7, wherein, The first convolutional network comprises a plurality of convolutional layers, and the process of obtaining the first convolutional feature comprises: Each convolutional layer in the plurality of convolutional layers is used to perform convolutional processing on the second encoded feature, to obtain a plurality of sub-convolutional features; The plurality of sub-convolutional features and the second encoded feature are subjected to normalization processing to obtain a plurality of normalized features; The normalized features are summed to obtain the first convolutional feature.

9. The method of claim 6, wherein, The dimension-reduced feature comprises a plurality of sub-features, each sub-feature having a different dimension, and the process of performing obstacle identification on the dimension-reduced feature to obtain obstacle information of the obstacle comprises: The last sub-feature in the dimension-reduced feature is subjected to deconvolution to obtain a deconvolutional feature having the same dimension as the second-to-last sub-feature; The deconvolutional feature and the last sub-feature are spliced to obtain a spliced feature; The spliced feature is subjected to obstacle identification to obtain the obstacle information of the obstacle.

10. An autonomous vehicle, comprising: The autonomous vehicle comprises one or more processors and one or more memories, the one or more memories storing at least one piece of program code, the at least one piece of program code being loaded and executed by the one or more processors to implement the operations performed by the obstacle identification method according to any one of claims 1 to 9.

Images