Point cloud label acquisition method, device, electronic device and autonomous driving vehicle

Abstract

The present disclosure provides a method, device, and autonomous driving vehicle for obtaining point cloud labels, in the field of artificial intelligence technology, specifically deep learning, semantic segmentation, and autonomous driving technology. The specific implementation scheme is as follows: obtaining M point cloud frames including the current point cloud frame, and projecting the point clouds in the M point cloud frames into the grids of the bird's-eye view, where M is an integer greater than or equal to 2; obtaining the target fusion features of the M point cloud frames corresponding to each grid based on the projected bird's-eye view; for each grid, obtaining the semantic segmentation label and dynamic and static labels of the grid based on the target fusion features of the grid; performing reverse projection on the grid to obtain the grid where the point cloud is located in the current point cloud frame, and determining the label of the grid where the point cloud is located as the label of the point cloud. The present disclosure performs feature fusion on multi-frame point clouds, obtains point cloud labels based on the fused features, reduces the time delay between obtaining point cloud labels, and improves the accuracy and reliability of point cloud labels.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, specifically deep learning, semantic segmentation, and autonomous driving, and in particular to a method and device for acquiring point cloud labels and an electronic device storage medium. Background Art

[0002] With the popularization and development of deep learning and lidar, it has gradually become possible to perform semantic segmentation and dynamic and static estimation of point cloud backgrounds through deep learning methods. In related technologies, the two tasks of semantic segmentation of point cloud backgrounds and dynamic and static estimation are independent of each other. When you want to obtain semantic segmentation results and dynamic and static estimation results at the same time, you need to use two deep learning models to generate semantic segmentation results and dynamic and static estimation results respectively. In the case of autonomous driving, the above methods often increase the latency of the autonomous driving perception link. Therefore, how to reduce the latency between obtaining the semantic segmentation label and dynamic and static label of each point cloud in the point cloud frame at the same time, while ensuring the accuracy and reliability of the label of each point cloud in the point cloud frame, has become an urgent problem to be solved. Summary of the Invention

[0003] The present disclosure provides a method, device, electronic device, storage medium, and program product for acquiring point cloud labels.

[0004] According to a first aspect, a method for obtaining point cloud labels is provided, comprising: obtaining M point cloud frames including a current point cloud frame, and projecting the point clouds in the M point cloud frames into grids of a bird's-eye view, respectively, where M is an integer greater than or equal to 2; obtaining target fusion features of the M point cloud frames corresponding to each grid based on the projected bird's-eye view; obtaining semantic segmentation labels and dynamic and static labels of each grid based on the target fusion features of the grid; performing reverse projection on each grid to determine the grid where the point cloud in the current point cloud frame is located, and determining the semantic segmentation labels and dynamic and static labels of the grid where the point cloud is located as the label of the point cloud.

[0005] According to the second aspect, a device for acquiring point cloud labels is provided, comprising: a projection module for acquiring M point cloud frames including a current point cloud frame, and projecting the point clouds in the M point cloud frames into grids of a bird's-eye view, respectively, where M is an integer greater than or equal to 2; a first acquisition module for acquiring target fusion features of the M point cloud frames corresponding to each grid based on the projected bird's-eye view; a second acquisition module for acquiring, for each grid, the semantic segmentation label and dynamic and static label of the grid based on the target fusion features of the grid; and a third acquisition module for reversely projecting each grid to determine the grid where the point cloud in the current point cloud frame is located, and determining the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for acquiring point cloud labels described in the first aspect of the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method for obtaining point cloud tags according to the first aspect of the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method for acquiring point cloud tags according to the first aspect of the present disclosure.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a flowchart of a method for obtaining point cloud tags according to the first embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of a method for acquiring point cloud tags according to the second embodiment of the present disclosure;

[0013] Figure 3 is a flowchart of a method for acquiring point cloud tags according to the third embodiment of the present disclosure;

[0014] Figure 4 is a schematic structural diagram of a backbone network according to the present disclosure;

[0015] Figure 5 is a flowchart of a method for obtaining point cloud tags according to the fourth embodiment of the present disclosure;

[0016] Figure 6 is a flowchart of a method for acquiring point cloud tags according to the fifth embodiment of the present disclosure;

[0017] Figure 7 is a schematic diagram of a method for obtaining point cloud labels according to the present disclosure;

[0018] Figure 8 is a block diagram of a device for acquiring point cloud labels for implementing an embodiment of the present disclosure;

[0019] Figure 9 It is a block diagram of an electronic device used to implement the method for obtaining point cloud tags according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0022] Deep learning (DL) is a new research direction in machine learning (ML). It was introduced to ML to bring it closer to its original goal: artificial intelligence. Deep learning studies the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound.

[0023] Semantic segmentation is a fundamental task in computer vision, in which visual input needs to be divided into different semantically interpretable categories, that is, the classification categories are meaningful in the real world.

[0024] Autonomous driving generally refers to automated driving systems, which utilize advanced communications, computing, networking, and control technologies to achieve real-time, continuous control of trains. Modern communication methods, directly connected to the train, enable two-way data communication between trains and the ground, with fast transmission rates and large amounts of information. This allows tracking trains and control centers to instantly determine the exact location of the preceding train, making operational management more flexible and control more efficient, making it more adaptable to the demands of automated train operation.

[0025] A method for obtaining point cloud labels according to an embodiment of the present disclosure is described below with reference to the accompanying drawings.

[0026] Figure 14 is a flowchart of a method for obtaining point cloud tags according to the first embodiment of the present disclosure.

[0027] like Figure 1 As shown, the method for obtaining point cloud tags in the embodiment of the present disclosure may specifically include the following steps:

[0028] S101 , obtaining M point cloud frames including the current point cloud frame, and projecting the point clouds in the M point cloud frames onto grids of a bird's-eye view, where M is an integer greater than or equal to 2.

[0029] Specifically, the execution entity of the point cloud tag acquisition method of the embodiment of the present disclosure may be the processing device provided in the embodiment of the present disclosure. The processing device may be a hardware device with data information processing capabilities and / or the necessary software to drive the operation of the hardware device. Optionally, the execution entity may include a workstation, server, computer, user terminal, and other devices. User terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals.

[0030] It should be noted that the present disclosure does not limit the specific method of obtaining the point cloud frame, and it can be selected according to actual conditions.

[0031] Optionally, a laser radar acquisition device can be used to obtain point cloud frames.

[0032] For example, point cloud frames can be acquired using image acquisition devices such as laser line scan cameras and binocular structured light cameras.

[0033] Optionally, a laser radar (Light Detection And Ranging, referred to as LiDAR) can be used to acquire point cloud frames.

[0034] Among them, the Bird's Eye View (BEV) is a three-dimensional picture drawn based on the principle of perspective, using high viewpoint perspective to look down at the undulating ground from a certain point at a high place.

[0035] It should be noted that projecting the point clouds in the M point cloud frames onto the grid of the bird's-eye view is to project the three-dimensional (x, y, z) coordinates of the point clouds onto the two-dimensional (x, y) coordinates of the grid.

[0036] It should be noted that the present disclosure does not limit the specific setting of the grid of the bird's-eye view, and it can be set according to actual conditions.

[0037] Optionally, 10m can be set as a grid; optionally, 20m can be set as a grid.

[0038] It should be noted that after obtaining the three-dimensional (x, y, z) coordinates of the point cloud, the point cloud can be projected into the grid corresponding to the bird's-eye view according to the three-dimensional (x, y, z) coordinates of the point cloud and the grid value of the bird's-eye view.

[0039] Optionally, when M is 2, the point clouds in the current point cloud frame and the previous point cloud frame may be projected into the grid of the bird's-eye view respectively.

[0040] S102 , obtaining target fusion features of M point cloud frames corresponding to each grid according to the projected bird's-eye view.

[0041] It should be noted that after projecting the point clouds in the M point cloud frames onto the grids of the bird's-eye view respectively, multiple initial feature information can be obtained based on the projected bird's-eye view, and then the multiple initial feature information can be processed to obtain the target fusion feature.

[0042] For example, based on the projected bird's-eye view, multiple initial features such as the number information of the point cloud, the reflectivity information of the point cloud, the height information of the point cloud, and the height difference information of the point cloud can be obtained. The multiple initial features are then spliced ​​together to obtain splicing features, and then feature extraction is performed on the splicing features to obtain target fusion features.

[0043] S103, based on the target fusion features of the grid, respectively obtain the semantic segmentation label and dynamic and static labels of the grid.

[0044] For example, after obtaining the target fusion features of the grid, the target fusion features can be input into the corresponding model to obtain the semantic segmentation label and dynamic and static labels of the grid respectively. The model has two output branches, which can simultaneously output the semantic segmentation label and dynamic and static labels of the grid.

[0045] S104 , performing reverse projection on each grid to determine the grid where the point cloud in the current point cloud frame is located, and determining the semantic segmentation label and the dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

[0046] In the disclosed embodiment, after obtaining the semantic segmentation label and dynamic and static labels of each grid, each grid can be back-projected. In other words, each grid is back-projected to the current point cloud frame, and the point cloud in the current point cloud frame covered by each grid can be determined, thereby determining the grid in which each point cloud in the current point cloud frame is located. Furthermore, the semantic segmentation label and dynamic and static labels of the grid in which the point cloud is located are determined as the label of the point cloud.

[0047] For example, for grid 1, when the semantic segmentation label of grid 1 is 1 and the dynamic and static label is 0, the semantic segmentation label of each point cloud in the current point cloud frame in grid 1 is 1 and the dynamic and static labels are 0.

[0048] In summary, the method for obtaining point cloud labels in the embodiment of the present disclosure obtains M point cloud frames including the current point cloud frame, and projects the point clouds in the M point cloud frames into the grids of the bird's-eye view, where M is an integer greater than or equal to 2. According to the projected bird's-eye view, the target fusion features of the M point cloud frames corresponding to each grid are obtained. For each grid, the semantic segmentation label and dynamic and static labels of the grid are obtained based on the target fusion features of the grid. Each grid is reversely projected to obtain the label of each point cloud in the current point cloud frame. The present disclosure projects the point clouds in multiple point cloud frames into the grid, and realizes the aggregation of point clouds in multiple point cloud frames through the grid, so that the same grid can have the features of multiple point cloud frames. Then, based on the target fusion features of the grid, the semantic segmentation label and dynamic and static labels of the grid are obtained. Then, the grid in which the point cloud in the current point cloud frame is located is determined by reverse projection, thereby determining the semantic segmentation label and dynamic and static labels of the point cloud in the current point cloud frame. This not only reduces the delay between obtaining the semantic segmentation label and dynamic and static labels of the point cloud, but also improves the accuracy and reliability of the point cloud label.

[0049] Figure 2 4 is a flow chart of a method for obtaining point cloud tags according to the second embodiment of the present disclosure.

[0050] like Figure 2 As shown, in Figure 1 Based on the illustrated embodiment, the method for obtaining point cloud tags in the embodiment of the present disclosure may specifically include the following steps:

[0051] S201 , obtaining M point cloud frames including the current point cloud frame, and projecting the point clouds in the M point cloud frames onto grids of the bird's-eye view, where M is an integer greater than or equal to 2.

[0052] Specifically, step S201 in this embodiment is the same as step S101 in the above embodiment, and will not be repeated here.

[0053] In the above embodiment, step S102 of “obtaining target fusion features of M point cloud frames corresponding to each grid according to the projected bird's-eye view” may specifically include the following steps S202 to S204 .

[0054] S202 : For each point cloud frame in the M point cloud frames, initial feature information of the point cloud frame corresponding to each grid is obtained according to the projected bird's-eye view of the point cloud frame.

[0055] As a possible implementation, Figure 3 As shown, based on the above embodiment, the specific process of obtaining the initial feature information of the point cloud frame corresponding to each grid according to the projected bird's-eye view of the point cloud frame in the above step S202 includes the following steps:

[0056] S301, obtaining a point cloud set located in each grid in the projected bird's-eye view.

[0057] Optionally, a point cloud set located in each grid in the projected bird's-eye view may be determined according to the two-dimensional (x, y) coordinates of the grid, wherein the point cloud set is composed of multiple point clouds.

[0058] S302: Determine initial feature information of each grid based on the point cloud set of each grid.

[0059] Optionally, the number of point clouds in the point cloud set, the height values ​​of the point clouds in the point cloud set can be obtained, and the height difference and / or average height of the point cloud set can be determined based on the height values, and the reflectivity of the point clouds in the point cloud set can be determined, and the average reflectivity of the point cloud set can be determined based on the reflectivity.

[0060] Optionally, the number of point clouds in the point cloud set, the height difference and / or average height of the point cloud set, and the average reflectivity of the point cloud set may be used as initial feature information of each grid.

[0061] S203 , stitching the initial feature information of each point cloud frame of the same grid to obtain candidate stitching features.

[0062] For example, the initial feature information of each point cloud frame of the same grid can be spliced ​​in the row dimension to obtain candidate splicing features.

[0063] S204 , processing the candidate splicing features of the grid through the backbone network to obtain the target fusion features of the grid.

[0064] Optionally, an initial backbone network structure can be established, the initial backbone network structure can be trained using a known data set and a validation set, and a total loss function can be set to supervise the initial backbone network structure to obtain a trained backbone network structure.

[0065] For example, the structure of the backbone network, such as Figure 4 As shown in the figure, the candidate concatenation features (concat) can be input into the backbone network, and the convolution layer and deconvolution layer of the backbone network process the candidate concatenation features to output the target fusion features.

[0066] In the above embodiment, step S103 of “obtaining semantic segmentation labels of the grid based on target fusion features of the grid” may specifically include the following steps S205 to S207 .

[0067] S205 , performing semantic segmentation on the target fusion features of the grid to obtain multiple semantic segmentation probabilities of the grid.

[0068] As a possible implementation, Figure 5As shown, based on the above embodiment, the specific process of performing semantic segmentation on the target fusion features of the grid and obtaining multiple semantic segmentation probabilities of the grid in the above step S205 includes the following steps:

[0069] S501, performing a first convolution process on the target fusion feature to obtain a first convolution fusion feature.

[0070] Optionally, the target fusion feature may be encoded using a two-dimensional convolution to obtain a fusion feature after the first convolution.

[0071] S502: Perform a first probability function mapping on the first convolution fusion feature to obtain multiple semantic segmentation probabilities.

[0072] Optionally, the first probability function may be a normalized exponential function, ie, a softmax function, through which a numerical vector may be normalized into a probability distribution vector, and the sum of the probabilities is one, ie, it may be mapped into a value of (0, 1).

[0073] For example, by performing softmax function mapping on the fused features after the first convolution, multiple semantic segmentation probabilities can be obtained.

[0074] S206 , determining a maximum semantic segmentation probability from the multiple semantic segmentation probabilities.

[0075] It should be noted that the present disclosure does not limit the specific method for determining the maximum semantic segmentation probability from multiple semantic segmentation probabilities, and it can be selected according to actual circumstances.

[0076] Optionally, the maximum semantic segmentation probability can be determined from multiple semantic segmentation probabilities by using the maximum independent variable point set argmax function.

[0077] S207 , determining the semantic segmentation label corresponding to the maximum semantic segmentation probability as the semantic segmentation label of the grid.

[0078] In the embodiment of the present disclosure, when determining the semantic segmentation label of the grid, the semantic segmentation label corresponding to the maximum semantic segmentation probability can be determined as the semantic segmentation label of the grid.

[0079] In the above embodiment, step S103 of “obtaining dynamic and static labels of the grid based on the target fusion features of the grid” may specifically include the following steps S208 to S209 .

[0080] S208, performing binary classification recognition on the target fusion features of the grid to obtain the type recognition probability of the grid.

[0081] As a possible implementation, Figure 6As shown, based on the above embodiment, the specific process of performing binary classification recognition on the target fusion features of the grid and obtaining the type recognition probability of the grid in the above step S208 includes the following steps:

[0082] S601, performing a second convolution process on the target fusion feature to obtain a second convolution fusion feature.

[0083] Optionally, the target fusion feature may be encoded using a two-dimensional convolution to obtain a second convolution-post-fusion feature.

[0084] S602, performing a second probability function mapping on the second convolution fusion feature to obtain the type recognition probability of the grid.

[0085] Optionally, the second probability function may be a sigmiod function, through which the variable may be mapped to a value of (0, 1).

[0086] For example, the sigmiod function mapping is performed on the fused features after the second convolution to obtain the type recognition probability of the grid.

[0087] S209 , comparing the type recognition probability with a preset probability threshold, and determining the dynamic and static labels of the grid based on the comparison result.

[0088] Optionally, the probability threshold may be preset to 0.5.

[0089] For example, if the type recognition probability is greater than 0.5, the dynamic and static labels of the grid may be determined to be 1; if the type recognition probability is less than 0.5, the dynamic and static labels of the grid may be determined to be 0.

[0090] S210 , performing reverse projection on each grid to obtain a label of each point cloud in the current point cloud frame.

[0091] Specifically, step S210 in this embodiment is the same as step S104 in the above embodiment, and will not be repeated here.

[0092] The present disclosure projects point clouds from multiple point cloud frames into a grid, and achieves aggregation of point clouds from multiple point cloud frames through the grid, so that the same grid can have the features of multiple point cloud frames, and then obtains the semantic segmentation label and dynamic and static labels of the grid based on the target fusion features of the grid, and then determines the grid where the point cloud in the current point cloud frame is located through reverse projection, thereby determining the semantic segmentation label and dynamic and static labels of the point cloud in the current point cloud frame, thereby improving the accuracy and reliability of the label of each point cloud in the point cloud frame. Furthermore, the present disclosure can simultaneously obtain the semantic segmentation label and dynamic and static labels of each point cloud in the point cloud frame through a complete neural network, reducing the delay between obtaining the semantic segmentation label and dynamic and static labels of the point cloud.

[0093] The following explains how to obtain point cloud labels.

[0094] For example, if Figure 7 As shown in the figure, the point cloud frame of the current frame and the point cloud frame of the previous frame can be projected onto the Bev grid to obtain the bird's-eye view after projection (Bev projection), and then the initial feature information of the point cloud frame corresponding to each grid, namely the handcraft feature (handcraft feature) feature), concatenate the initial feature information generated by the two frames of point cloud, and input them into the model, wherein the model includes a backbone network, a semantic segmentation network, and a dynamic and static estimation network. Optionally, an initial backbone network structure can be established, and the initial backbone network structure can be trained using a known data set and a validation set, and a total loss function is set to supervise the initial backbone network structure to obtain a trained backbone network structure. Optionally, an initial semantic segmentation network structure can be established, and the initial semantic segmentation network structure can be trained using a known data set and a validation set, and a total loss function is set to supervise the initial semantic segmentation network structure to obtain a trained semantic segmentation network structure. Optionally, an initial dynamic and static estimation network structure can be established, and the initial dynamic and static estimation network structure can be trained using a known data set and a validation set, and a total loss function is set to supervise the initial dynamic and static estimation network structure to obtain a trained dynamic and static estimation network structure. First, , the candidate splicing features are downsampled through the backbone network to obtain the target fusion features (Deconv2d, s=4). In the output branch of semantic segmentation, the target fusion features are encoded using two-dimensional convolution, and the output is converted into a probability value using the softmax function. The maximum semantic segmentation probability is determined from multiple semantic segmentation probabilities through the argmax function, and the semantic segmentation label corresponding to the maximum semantic segmentation probability is determined as the semantic segmentation label of the grid. In the output branch of dynamic and static estimation, the target fusion features are encoded using two-dimensional convolution, and then the output is converted into a probability value using the sigmiod function. By setting a probability threshold of 0.5, the type recognition probability is compared with the preset probability threshold, and the dynamic and static labels of the grid are determined based on the comparison results. After obtaining the semantic segmentation labels and dynamic and static labels in the bev grid, the label of each point cloud in the current frame point cloud frame can be obtained by back projection.

[0095] The present disclosure projects point clouds from multiple point cloud frames into a grid, and achieves aggregation of point clouds from multiple point cloud frames through the grid, so that the same grid can have the features of multiple point cloud frames, and then obtains the semantic segmentation label and dynamic and static labels of the grid based on the target fusion features of the grid, and then determines the grid where the point cloud in the current point cloud frame is located through reverse projection, thereby determining the semantic segmentation label and dynamic and static labels of the point cloud in the current point cloud frame, thereby improving the accuracy and reliability of the label of each point cloud in the point cloud frame. Furthermore, the present disclosure can simultaneously obtain the semantic segmentation label and dynamic and static labels of each point cloud in the point cloud frame through a complete neural network, reducing the delay between obtaining the semantic segmentation label and dynamic and static labels of the point cloud.

[0096] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0097] Figure 8 3 is a schematic structural diagram of a device for acquiring point cloud tags according to an embodiment of the present disclosure.

[0098] like Figure 8 As shown, the point cloud tag acquisition device 800 includes: a projection module 810, a first acquisition module 820, a second acquisition module 830 and a third acquisition module 840.

[0099] A projection module 810 is configured to obtain M point cloud frames including the current point cloud frame, and project the point clouds in the M point cloud frames onto a grid of the bird's-eye view, where M is an integer greater than or equal to 2;

[0100] A first acquisition module 820 is configured to acquire target fusion features of the M point cloud frames corresponding to each grid according to the projected bird's-eye view;

[0101] The second acquisition module 830 is used to obtain, for each grid, a semantic segmentation label and a dynamic and static label of the grid based on the target fusion features of the grid;

[0102] The third acquisition module 840 is used to perform reverse projection on each grid, determine the grid where the point cloud in the current point cloud frame is located, and determine the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

[0103] The first acquisition module 820 is further configured to:

[0104] For each of the M point cloud frames, obtaining initial feature information of the point cloud frame corresponding to each grid according to the projected bird's-eye view of the point cloud frame;

[0105] The initial feature information of each point cloud frame of the same grid is spliced ​​to obtain candidate splicing features;

[0106] The candidate splicing features are processed through a backbone network to obtain the target fusion features.

[0107] The first acquisition module 820 is further configured to:

[0108] Obtain a point cloud set located in each grid in the projected bird's-eye view image;

[0109] Based on the point cloud set of each grid, the initial feature information of each grid is determined.

[0110] The first acquisition module 820 is further configured to:

[0111] Obtain the number of point clouds in the point cloud set;

[0112] Acquire height values ​​of point clouds in the point cloud set, and determine a height difference and / or an average height of the point cloud set according to the height values;

[0113] Obtain reflectivity of point clouds in the point cloud set, and determine an average reflectivity of the point cloud set based on the reflectivity.

[0114] The second acquisition module 830 is further configured to:

[0115] For each grid, performing semantic segmentation on the target fusion features of the grid to obtain multiple semantic segmentation probabilities of the grid;

[0116] Determining a maximum semantic segmentation probability from the multiple semantic segmentation probabilities;

[0117] The semantic segmentation label corresponding to the maximum semantic segmentation probability is determined as the semantic segmentation label of the grid.

[0118] The second acquisition module 830 is further configured to:

[0119] Performing a first convolution process on the target fusion feature to obtain a first convolution-post-fusion feature;

[0120] Perform a first probability function mapping on the first convolution fusion feature to obtain the multiple semantic segmentation probabilities.

[0121] The second acquisition module 830 is further configured to:

[0122] For each grid, perform binary classification recognition on the target fusion features of the grid to obtain the type recognition probability of the grid;

[0123] The type recognition probability is compared with a preset probability threshold, and the dynamic and static labels of the grid are determined based on the comparison result.

[0124] The second acquisition module 830 is further configured to:

[0125] Performing a second convolution process on the target fusion feature to obtain a second convolution fusion feature;

[0126] A second probability function mapping is performed on the second convolution fusion feature to obtain the type recognition probability of the grid.

[0127] It should be noted that the above explanation of the embodiment of the method for obtaining point cloud tags is also applicable to the device for obtaining point cloud tags in the embodiment of the present disclosure, and the specific process will not be repeated here.

[0128] The present disclosure projects point clouds from multiple point cloud frames into a grid, and achieves aggregation of point clouds from multiple point cloud frames through the grid, so that the same grid can have the features of multiple point cloud frames, and then obtains the semantic segmentation label and dynamic and static labels of the grid based on the target fusion features of the grid, and then determines the grid where the point cloud in the current point cloud frame is located through reverse projection, thereby determining the semantic segmentation label and dynamic and static labels of the point cloud in the current point cloud frame, thereby improving the accuracy and reliability of the label of each point cloud in the point cloud frame. Furthermore, the present disclosure can simultaneously obtain the semantic segmentation label and dynamic and static labels of each point cloud in the point cloud frame through a complete neural network, reducing the delay between obtaining the semantic segmentation label and dynamic and static labels of the point cloud.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 907 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0132] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 901 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the method for obtaining point cloud labels. For example, in some embodiments, the method for obtaining point cloud labels can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the model training or point cloud label acquisition method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the method for acquiring point cloud tags in any other appropriate manner (for example, by means of firmware).

[0134] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0139] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0140] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for obtaining point cloud tags as described above.

[0141] The present disclosure also provides an autonomous vehicle, which may include the electronic device described in the above embodiment, and which is configured to execute the method for acquiring point cloud labels described in the above embodiment. The autonomous vehicle is provided with a point cloud acquisition device, which acquires point cloud frames. The acquired point cloud frames may be input into the electronic device, which then executes the method for acquiring point cloud labels described in the above embodiment on the input point cloud frames.

[0142] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0143] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for obtaining point cloud labels, wherein: include: Acquire M point cloud frames including the current point cloud frame, and project the point clouds in the M point cloud frames onto the grids of the bird's-eye view, where M is an integer greater than or equal to 2; For each of the M point cloud frames, obtaining initial feature information of the point cloud frame corresponding to each grid according to the projected bird's-eye view of the point cloud frame; The initial feature information of each point cloud frame of the same grid is spliced ​​to obtain candidate splicing features; For each grid, the candidate splicing features of the grid are processed by the backbone network in the model to obtain the target fusion features of the grid; For each grid, the target fusion features of the grid are input into the semantic segmentation network and the dynamic and static estimation network in the model to obtain the semantic segmentation label and dynamic and static label of the grid; wherein the model includes a backbone network, a semantic segmentation network and a dynamic and static estimation network, and the semantic segmentation network and the dynamic and static estimation network are two output branches of the model to simultaneously output the semantic segmentation label and dynamic and static label of the grid; Back-projecting is performed on each grid to determine the grid where the point cloud in the current point cloud frame is located, and the semantic segmentation label and the dynamic and static label of the grid where the point cloud is located are determined as the label of the point cloud.

2. The method according to claim 1, wherein The step of obtaining initial feature information of the point cloud frame corresponding to each grid according to the projected bird's-eye view of the point cloud frame includes: Obtain a point cloud set located in each grid in the projected bird's-eye view image; Based on the point cloud set of each grid, the initial feature information of each grid is determined.

3. The method according to claim 2, wherein: The determining of the initial feature information of each grid based on the point cloud set of each grid includes: Obtain the number of point clouds in the point cloud set; Acquire height values ​​of point clouds in the point cloud set, and determine a height difference and / or an average height of the point cloud set according to the height values; Obtain reflectivity of point clouds in the point cloud set, and determine an average reflectivity of the point cloud set based on the reflectivity.

4. The method according to any one of claims 1 to 3, wherein For each grid, based on the target fusion features of the grid, a semantic segmentation label of the grid is obtained, including: Performing semantic segmentation on the target fusion features of the grid to obtain multiple semantic segmentation probabilities of the grid; Determining a maximum semantic segmentation probability from the multiple semantic segmentation probabilities; The semantic segmentation label corresponding to the maximum semantic segmentation probability is determined as the semantic segmentation label of the grid.

5. The method according to claim 4, wherein The performing semantic segmentation on the target fusion features of the grid to obtain multiple semantic segmentation probabilities of the grid includes: Performing a first convolution process on the target fusion feature to obtain a first convolution-post-fusion feature; Perform a first probability function mapping on the first convolution fusion feature to obtain the multiple semantic segmentation probabilities.

6. The method according to any one of claims 1 to 3, wherein For each grid, based on the target fusion features of the grid, the dynamic and static labels of the grid are obtained, including: Performing binary classification recognition on the target fusion features of the grid to obtain the type recognition probability of the grid; The type recognition probability is compared with a preset probability threshold, and the dynamic and static labels of the grid are determined based on the comparison result.

7. The method according to claim 6, wherein: The performing binary classification recognition on the target fusion features of the grid to obtain the type recognition probability of the grid includes: Performing a second convolution process on the target fusion feature to obtain a second convolution fusion feature; A second probability function mapping is performed on the second convolution fusion feature to obtain the type recognition probability of the grid.

8. A device for acquiring point cloud labels, wherein: include: a projection module, configured to obtain M point cloud frames including the current point cloud frame, and project the point clouds in the M point cloud frames onto the grids of the bird's-eye view, where M is an integer greater than or equal to 2; A first acquisition module is configured to obtain, for each of the M point cloud frames, initial feature information of the point cloud frame corresponding to each grid based on a projected bird's-eye view of the point cloud frame; splice the initial feature information of each point cloud frame of the same grid to obtain candidate splicing features; and process the candidate splicing features of each grid through the backbone network in the model to obtain a target fusion feature of the grid; A second acquisition module is configured to input the target fusion features of each grid into the semantic segmentation network and the dynamic and static estimation network in the model to obtain the semantic segmentation label and dynamic and static label of the grid, respectively; wherein the model includes a backbone network, a semantic segmentation network, and a dynamic and static estimation network, and the semantic segmentation network and the dynamic and static estimation network are two output branches of the model to simultaneously output the semantic segmentation label and dynamic and static label of the grid; The third acquisition module is used to perform reverse projection on each grid, determine the grid where the point cloud in the current point cloud frame is located, and determine the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

9. The device according to claim 8, wherein The first acquisition module is further configured to: Obtain a point cloud set located in each grid in the projected bird's-eye view image; Based on the point cloud set of each grid, the initial feature information of each grid is determined.

10. The device according to claim 9, characterized in that The first acquisition module is further configured to: Obtain the number of point clouds in the point cloud set; Acquire height values ​​of point clouds in the point cloud set, and determine a height difference and / or an average height of the point cloud set according to the height values; Obtain reflectivity of point clouds in the point cloud set, and determine an average reflectivity of the point cloud set based on the reflectivity.

11. The device according to any one of claims 8 to 10, wherein: The second acquisition module is further configured to: For each grid, performing semantic segmentation on the target fusion features of the grid to obtain multiple semantic segmentation probabilities of the grid; Determining a maximum semantic segmentation probability from the multiple semantic segmentation probabilities; The semantic segmentation label corresponding to the maximum semantic segmentation probability is determined as the semantic segmentation label of the grid.

12. The device according to claim 11, wherein The second acquisition module is further configured to: Performing a first convolution process on the target fusion feature to obtain a first convolution-post-fusion feature; Perform a first probability function mapping on the first convolution fusion feature to obtain the multiple semantic segmentation probabilities.

13. The device according to any one of claims 8 to 10, characterized in that The second acquisition module is further configured to: For each grid, perform binary classification recognition on the target fusion features of the grid to obtain the type recognition probability of the grid; The type recognition probability is compared with a preset probability threshold, and the dynamic and static labels of the grid are determined based on the comparison result.

14. The device according to claim 13, wherein The second acquisition module is further configured to: Performing a second convolution process on the target fusion feature to obtain a second convolution fusion feature; A second probability function mapping is performed on the second convolution fusion feature to obtain the type recognition probability of the grid.

15. An electronic device, wherein: including processor and memory; The processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method according to claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claims 1 to 7 is implemented.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to claims 1-7.

18. An autonomous driving vehicle comprising the electronic device according to claim 15.

Images