ASIC hardware acceleration and end-to-end laser point cloud implementation method and system

By designing the ASIC hardware accelerator, combining the 3D sparse convolution algorithm and end-to-end algorithm of the NPU chip, the problems of delay and resource consumption in 3D point cloud data processing are solved, and efficient laser point cloud processing is achieved.

CN119027591APending Publication Date: 2024-11-26SIENGINE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411167796.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When processing 3D point cloud data on CPU and GPU-based hardware, the prior art has problems such as long delay, large memory space and low MAC utilization, especially when implementing 3D convolution on NPU hardware, it is inefficient.

Method used

Design the ASIC hardware accelerator to perform hardware mapping through hardware acceleration, dynamic voxelization and voxel feature extraction, combining 3D sparse convolution algorithm based on NPU chips and end-to-end algorithms of voxel or pillar features to achieve hardware acceleration and End-to-end laser point cloud processing.

Benefits of technology

It effectively solves the delay problem of LiDAR point cloud preprocessing algorithm, reduces the calculation time and resource consumption of 3D convolution on the end-side hardware, and improves the MAC utilization rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027591A_ABST
    Figure CN119027591A_ABST
Patent Text Reader

Abstract

The invention discloses an ASIC hardware acceleration and end-to-end laser point cloud implementation method and system, and relates to the technical field of hardware circuit acceleration, and the method comprises the steps: designing an ASIC of laser point cloud pre-processing coding, and obtaining a LiDAR point cloud ASA, and the LiDAR point cloud ASA comprises hard voxelization, dynamic voxelization and voxel feature extraction; hardware mapping is carried out through a 3D sparse convolution algorithm based on an NPU chip; and hardware mapping is carried out through an end-to-end algorithm based on voxel or pilar features, so that hardware acceleration and end-to-end laser point cloud implementation are realized. According to the method, the problem of delay of a LiDAR point cloud preprocessing algorithm is solved through hardware ASIC acceleration, and meanwhile, the problems of long consumed time and more consumed resources of 3D convolution on end side hardware can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of hardware circuit acceleration, and in particular to an ASIC hardware acceleration and end-to-end laser point cloud implementation method and system. Background Art

[0002] With the continuous development of 3D (3 Dimension) point cloud neural networks, 3D point cloud neural networks have become an important method for processing sparse 3D point cloud data in the fields of autonomous driving, robotics, virtual reality, etc. For 3D point cloud laser pre-processing, such as Voxel encoding and Pillar encoding, they are all processed in parallel on the CPU (Central Processing Unit) side or using CUDA (a parallel computing platform and programming model); for 3D sparse convolution, such as Spconv1 and Spconv2, they are 3D point cloud acceleration libraries designed specifically for CUDA. Among them, Voxel is a voxel, which divides the 3D space scanned by the laser into small cubic spaces. Each space and the laser point inside the space are called voxels; Pillar divides the 3D space scanned by the laser into small cylindrical spaces with the ground plane as the bottom surface. Spconv, the full name of Sparse Convolution, is sparse convolution.

[0003] However, due to the large number of point clouds (up to millions), for voxel / Pillar-based encoding methods, point clouds are encoded using the CPU, which has low computational efficiency on AI (Artificial Intelligence) chips based on NPU (embedded neural network processor), resulting in long network delays for end-to-end lidar algorithms. At the same time, when performing sparse 3D convolution on the Voxel after point cloud encoding, existing optimization algorithms (such as Spconv 1.x and 2.x) are on the GPU (Graphics Processing Unit) side. Implementing the algorithm on NPU-based hardware requires large memory space, long time consumption, and low MAC (Media Access Control) utilization. Summary of the invention

[0004] The present application provides an ASIC hardware acceleration and end-to-end laser point cloud implementation method and system, which solves the delay problem of the LiDAR point cloud pre-processing algorithm through hardware ASIC acceleration. At the same time, it can also solve the problem that 3D convolution is time-consuming and consumes a lot of resources on the end-side hardware.

[0005] In a first aspect, an embodiment of the present application provides an ASIC hardware acceleration and end-to-end laser point cloud implementation method, wherein the ASIC hardware acceleration and end-to-end laser point cloud implementation method comprises: Design an ASIC for laser point cloud pre-processing encoding to obtain a LiDAR point cloud ASA, wherein the LiDAR point cloud ASA includes hard voxelization, dynamic voxelization, and voxel feature extraction; Hardware mapping is performed through the 3D sparse convolution algorithm based on the NPU chip; Through end-to-end algorithms based on voxel or pillar features, hardware mapping is performed to achieve hardware acceleration and end-to-end laser point cloud realization.

[0006] In combination with the first aspect, in one implementation, The dynamic voxelization is a process of converting continuous point cloud data into discrete voxel representation; The hard voxelization is to aggregate the point cloud data into voxels based on the dynamic voxelization; The voxel feature extraction is to take points or padding, perform feature encoding on the point cloud, and AISC performs parallel calculations on different voxels or Pillars.

[0007] In combination with the first aspect, in one implementation, the hardware mapping is performed by using a 3D sparse convolution algorithm based on an NPU chip, specifically including: After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain sparse representation; After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain a collection table; According to the sparse representation and the collection table, the collected features are obtained, and the obtained collected features are multiplied by the expanded convolution kernel.

[0008] In combination with the first aspect, in one implementation, For sparse representation, specifically, after the 3D space scanned by the laser is divided into multiple Voxels, the entire Voxel is constructed into a sparse representation, and the directory of the sparse representation includes entries id, pos, and values, wherein the pos is the position of the Voxel in the 3D voxel space, and the values ​​is the feature vector of the Voxel, wherein the position where the data in the Voxel is not 0 is constructed as an entry in the sparse representation; For the collection table, specifically, the collection table is used to arrange the positions where convolution calculations need to be performed. The directory of the collection table includes a position id and a digital number. The position id represents the id of the position where convolution is required, and the digital number is the Voxel that needs to be calculated corresponding to each 3D convolution kernel.

[0009] In combination with the first aspect, in one implementation, when the collected features are multiplied by the expanded convolution kernel: The dimension of the expanded convolution kernel is expressed as (IC * K) * OC, where IC is the input feature dimension of the convolution kernel of a certain layer, OC is the output feature dimension of the convolution kernel of a certain layer, and K is the number of 3D convolution kernels; The dimension of the collected features is expressed as N*IC*K, where N is the total length of the position id in the collection table.

[0010] In combination with the first aspect, in one implementation, the hardware mapping is performed by an end-to-end algorithm based on voxel or pillar features, specifically including: Send the point cloud to ASA to extract Voxel and Pillar features to obtain Voxel features and Pillar features; The Pillar features are fed into a 2D convolutional network for feature extraction, and the Voxel features are fed into a sparse 3D convolutional network for feature extraction; The features extracted by the sparse 3D convolutional network are sent to the Neck or Head based on the convolutional network, or to the Neck or Head based on the Transformer, and the output result is obtained and post-processed to obtain the final output of the end-to-end algorithm.

[0011] In combination with the first aspect, in one implementation, When extracting Pillar features, the corresponding implementation hardware is NPU; When extracting Voxel features, the corresponding implementation hardware is NPU, DSP or SCU.

[0012] In a second aspect, an embodiment of the present application provides an ASIC hardware acceleration and end-to-end laser point cloud implementation system, wherein the ASIC hardware acceleration and end-to-end laser point cloud implementation system comprises: A design module, which is used to design an ASIC for laser point cloud pre-processing coding to obtain a LiDAR point cloud ASA, wherein the LiDAR point cloud ASA includes hard voxelization, dynamic voxelization and voxel feature extraction; A first execution module, which is used to perform hardware mapping through a 3D sparse convolution algorithm based on an NPU chip; The second execution module is used to perform hardware mapping through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud realization.

[0013] In conjunction with the second aspect, in one implementation, The dynamic voxelization is a process of converting continuous point cloud data into discrete voxel representation; The hard voxelization is to aggregate the point cloud data into voxels based on the dynamic voxelization; The voxel feature extraction is to take points or padding, perform feature encoding on the point cloud, and AISC performs parallel calculations on different voxels or Pillars.

[0014] In conjunction with the second aspect, in one implementation, the hardware mapping is performed by using a 3D sparse convolution algorithm based on an NPU chip, specifically including: After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain sparse representation; After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain a collection table; According to the sparse representation and the collection table, the collected features are obtained, and the obtained collected features are multiplied by the expanded convolution kernel.

[0015] The beneficial effects brought by the technical solution provided by the embodiments of the present application include: By designing an ASIC for laser point cloud pre-processing encoding, a LiDAR point cloud ASA is obtained, which includes hardware voxelization, dynamic voxelization and voxel feature extraction. Then, hardware mapping is performed through a 3D sparse convolution algorithm based on an NPU chip. Then, hardware mapping is performed through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud implementation. The delay problem of the LiDAR point cloud pre-processing algorithm is solved through hardware ASIC acceleration. At the same time, for the calculation method of 3D sparse convolution, an algorithm flow based on Gather Table and Sparse Representation is designed and hardware mapping is performed to make it more suitable for end-side chips, solving the problem of long time and high resource consumption of 3D convolution on end-side hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the ASIC hardware acceleration and end-to-end laser point cloud implementation method for this application; Figure 2 This is a schematic diagram of the laser point cloud pre-processing ASIC; Figure 3 It is a schematic diagram of the overall process of the 3D sparse convolution algorithm; Figure 4 Schematic diagram of hardware mapping of end-to-end algorithms based on voxel or pillar features; Figure 5This is a functional module diagram of the ASIC hardware acceleration and end-to-end laser point cloud implementation system for this application; Figure 6 Schematic diagram of the hardware structure of the ASIC hardware acceleration and end-to-end laser point cloud implementation equipment for this application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0019] On the first aspect, the embodiment of the present application provides an ASIC hardware acceleration and end-to-end laser point cloud implementation method, which solves the delay problem of LiDAR (a system that integrates three technologies of laser, global positioning system and inertial navigation system, used to obtain point cloud data and generate accurate digital three-dimensional models) point cloud pre-processing algorithm through hardware ASIC (Application Specific Integrated Circuit) acceleration. At the same time, for the calculation method of 3D sparse convolution, an algorithm flow based on Gather Table and Sparse Representation is designed and hardware mapping is performed to make it more suitable for the end-side chip (NPU), so as to solve the problem of long time and high resource consumption of 3D convolution on the end-side hardware.

[0020] In one embodiment, referring to Figure 1 , Figure 1 This is a flowchart of the ASIC hardware acceleration and end-to-end laser point cloud implementation method for this application. Figure 1 As shown, the ASIC hardware acceleration and end-to-end laser point cloud implementation method includes: S1: Design an ASIC for laser point cloud pre-processing encoding, namely, LiDAR point cloud ASA, which includes hard voxelization, dynamic voxelization and voxel feature extraction; See also Figure 2The figure shows a schematic diagram of the laser point cloud pre-processing ASIC, where the LiDAR point cloud ASA is an ASIC designed for laser point cloud pre-processing encoding. The LiDAR point cloud ASA is divided into three parts: hard voxelization, dynamic voxelization, and voxel features. DDR stands for Double Rate Synchronous Dynamic Random Access Memory.

[0021] For dynamic voxelization, dynamic voxelization is the process of converting continuous point cloud data into discrete voxel representation. Traverse each point cloud, calculate the coordinates of the point cloud in the voxel space according to the coordinates and voxel parameters of the point cloud, and ASIC first divides the entire LiDAR space. For example, the LiDAR space is divided into N parts, and the number of parallel modules of hard voxelization + dynamic voxelization is N.

[0022] For hard voxelization, hard voxelization is to aggregate point cloud data into voxels based on dynamic voxelization.

[0023] For voxel feature extraction, voxel feature extraction is point extraction or padding, feature encoding of point cloud, and AISC performs parallel calculations on different voxels or Pillars. For example, if all voxels / pillars are divided into M parts, the number of voxel feature extraction modules is M.

[0024] S2: Hardware mapping is performed through the 3D sparse convolution algorithm based on the NPU chip; See also Figure 3 As shown, it is a schematic diagram of the overall process of the 3D sparse convolution algorithm. In this application, hardware mapping is performed through the 3D sparse convolution algorithm based on the NPU chip, which specifically includes: S201: After the point cloud is subjected to feature extraction by ASA (Algorithm Specific Accelerator), it is processed by CPU, DSP (Digital Signal Processing) or SCU (Sparse Convolution Unit) to obtain sparse representation; For sparse representation, specifically, after the 3D space scanned by the laser is divided into multiple Voxels, there are no laser points in most Voxels. At this time, in order to obtain a denser representation, the entire Voxel is constructed into a sparse representation. As shown in Table 1 below, it is a table diagram of sparse representation. The directory of sparse representation includes entries id, pos, and values. Pos is the position of Voxel in 3D voxel space (x, y, z), and the values ​​is the feature vector of Voxel. The dimension of the vector is not fixed and can be designed by the algorithm personnel. Among them, the position where the data in Voxel is not 0 is constructed as an entry in the sparse representation; Table 1 Tabular representation of sparse representation

[0025] S202: After extracting features from the point cloud through ASA, the point cloud is processed by the CPU, DSP or SCU to obtain a collection table; For the collection table, specifically, the collection table is used to arrange the positions that need to be convolved. See Table 2 below, which is a table diagram of the collection table. The directory of the collection table includes a position id and a digital number. The position id represents the id of the position that needs to be convolved, and the digital number is the Voxel that needs to be calculated corresponding to each 3D convolution kernel. 0~26 in Table 2 represents the Voxel that needs to be calculated corresponding to 27 3D convolution kernels, " / " represents an empty Voxel, and 1 and 2 in 0~26 correspond to the entries corresponding to the entry id in the sparse representation of Table 1.

[0026] Table 2 Collection table diagram

[0027] S203: Obtain collected features according to the sparse representation and the collection table, and multiply the obtained collected features by the expanded convolution kernel.

[0028] It should be noted that if the width of the convolution kernel is 3, the dimension of the convolution kernel of the conventional 3D convolution is IC * 3*3*3*OC. In this application, when the collected features are multiplied by the expanded convolution kernel: the dimension of the expanded convolution kernel is expressed as (IC* K) * OC, where IC, i.e. Input Channel, is the input feature dimension of the convolution kernel of a certain layer, OC, i.e. Output Channel, is the output feature dimension of the convolution kernel of a certain layer, and K is the number of 3D convolution kernels. For example, if the 3D convolution kernel is (3*3*3), then K = 27; in this application, the dimension of the collected features is expressed as N*IC*K, where N is the total length of the position id in the collection table.

[0029] Figure 3 In the figure, Sparse Representation represents sparse representation, Gather Table represents collection table, Gathered Feature represents the collected features, and Coefficient Tensor represents the expanded convolution kernel.

[0030] S3: Hardware mapping is performed through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud implementation. This application maps the process of the end-to-end algorithm of the laser point cloud on the NPU chip. After the original point cloud is pre-processed by sampling and splicing, the end-to-end algorithm based on voxel or pillar features is used for hardware mapping.

[0031] See also Figure 4 FIG. 1 is a schematic diagram of hardware mapping of an end-to-end algorithm based on voxel or pillar features. In this application, hardware mapping is performed by an end-to-end algorithm based on voxel or pillar features, specifically including: S301: Send the point cloud to ASA to extract Voxel and Pillar features to obtain Voxel features and Pillar features; S302: Send the Pillar feature to a 2D convolutional network for feature extraction, and send the Voxel feature to a sparse 3D convolutional network for feature extraction; that is, for the Pillar feature, send it to a 2D convolutional network for feature extraction, and the corresponding implementation hardware is an NPU; for the Voxel feature, send it to a 3D convolutional network for feature extraction, and the corresponding implementation hardware is an NPU, DSP or SCU; S303: The features obtained after being extracted by the sparse 3D convolutional network are sent to the Neck (neck network) or Head (head network) based on the convolutional network, or to the Neck or Head based on Transformer (a deep learning model architecture for natural language processing and other sequence-to-sequence tasks), and the output result is obtained and post-processed to obtain the final output of the end-to-end algorithm. The network output result is post-processed to obtain the final output of the algorithm, and the post-processing hardware unit is mainly a CPU or DSP.

[0032] Figure 4 In the figure, 3D Sparse Conv represents sparse 3D convolutional network, 2D Conv represents 2D convolutional network, Conv represents convolutional network, and Postprocess represents post-processing.

[0033] The ASIC hardware acceleration and end-to-end laser point cloud implementation method of the embodiment of the present application obtains a LiDAR point cloud ASA by designing an ASIC for laser point cloud pre-processing encoding. The LiDAR point cloud ASA includes hardware voxelization, dynamic voxelization and voxel feature extraction, and then performs hardware mapping through a 3D sparse convolution algorithm based on an NPU chip, and then performs hardware mapping through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud implementation, and solving the delay problem of the LiDAR point cloud pre-processing algorithm through hardware ASIC acceleration. At the same time, for the calculation method of 3D sparse convolution, an algorithm flow based on Gather Table and Sparse Representation is designed and hardware mapping is performed to make it more suitable for end-side chips, thereby solving the problem of 3D convolution being time-consuming and consuming a lot of resources on end-side hardware.

[0034] In a second aspect, the embodiments of the present application also provide an ASIC hardware acceleration and end-to-end laser point cloud implementation system.

[0035] In one embodiment, referring to Figure 5 , Figure 5 This is a functional module diagram of the ASIC hardware acceleration and end-to-end laser point cloud implementation system for this application. Figure 5 As shown, the ASIC hardware acceleration and end-to-end laser point cloud implementation system includes: a design module, a first execution module, and a second execution module.

[0036] The design module is used to design an ASIC for laser point cloud pre-processing encoding to obtain a LiDAR point cloud ASA, which includes hardware voxelization, dynamic voxelization and voxel feature extraction; the first execution module is used to perform hardware mapping through a 3D sparse convolution algorithm based on an NPU chip; the second execution module is used to perform hardware mapping through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud implementation.

[0037] In the present application, dynamic voxelization is the process of converting continuous point cloud data into discrete voxel representation; the hard voxelization is to aggregate the point cloud data into voxels based on dynamic voxelization; the voxel feature extraction is to take points or padding to perform feature encoding on the point cloud, and AISC performs parallel calculations on different voxels or Pillars.

[0038] In this application, hardware mapping is performed through a 3D sparse convolution algorithm based on an NPU chip, specifically including: After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain sparse representation; After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain a collection table; According to the sparse representation and the collection table, the collected features are obtained, and the obtained collected features are multiplied by the expanded convolution kernel.

[0039] Among them, the functional implementation of each module in the above-mentioned ASIC hardware acceleration and end-to-end laser point cloud implementation system corresponds to the steps in the above-mentioned ASIC hardware acceleration and end-to-end laser point cloud implementation method embodiment, and its functions and implementation process will not be repeated here one by one.

[0040] In a third aspect, an embodiment of the present application provides an ASIC hardware acceleration and end-to-end laser point cloud implementation device. The ASIC hardware acceleration and end-to-end laser point cloud implementation device can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0041] Reference Figure 6 , Figure 6 The hardware structure diagram of the ASIC hardware acceleration and end-to-end laser point cloud implementation device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the ASIC hardware acceleration and end-to-end laser point cloud implementation device may include a processor, a memory, a communication interface, and a communication bus.

[0042] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0043] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the ASIC hardware acceleration and end-to-end laser point cloud realization device, and the interface used to realize the interconnection of the ASIC hardware acceleration and end-to-end laser point cloud realization device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0044] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0045] The processor may be a general-purpose processor, which may call the ASIC hardware acceleration and end-to-end laser point cloud implementation program stored in the memory, and execute the ASIC hardware acceleration and end-to-end laser point cloud implementation method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the ASIC hardware acceleration and end-to-end laser point cloud implementation program is called may refer to the various embodiments of the ASIC hardware acceleration and end-to-end laser point cloud implementation method of the present application, which will not be repeated here.

[0046] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0047] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0048] The computer-readable storage medium of the present application stores an ASIC hardware acceleration and end-to-end laser point cloud implementation program, wherein when the ASIC hardware acceleration and end-to-end laser point cloud implementation program is executed by a processor, the steps of the above-mentioned ASIC hardware acceleration and end-to-end laser point cloud implementation method are implemented.

[0049] Among them, the methods implemented when the ASIC hardware acceleration and end-to-end laser point cloud implementation program is executed can refer to the various embodiments of the ASIC hardware acceleration and end-to-end laser point cloud implementation method of the present application, and will not be repeated here.

[0050] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0051] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0052] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0053] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0054] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0056] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An ASIC hardware acceleration and end-to-end laser point cloud implementation method, characterized in that: The ASIC hardware acceleration and end-to-end laser point cloud implementation method includes: Design an ASIC for laser point cloud pre-processing encoding to obtain a LiDAR point cloud ASA, wherein the LiDAR point cloud ASA includes hard voxelization, dynamic voxelization, and voxel feature extraction; Hardware mapping is performed through the 3D sparse convolution algorithm based on the NPU chip; Through end-to-end algorithms based on voxel or pillar features, hardware mapping is performed to achieve hardware acceleration and end-to-end laser point cloud realization.

2. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 1, characterized in that: The dynamic voxelization is a process of converting continuous point cloud data into discrete voxel representation; The hard voxelization is to aggregate the point cloud data into voxels based on the dynamic voxelization; The voxel feature extraction is to take points or padding, perform feature encoding on the point cloud, and AISC performs parallel calculations on different voxels or Pillars.

3. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 1, characterized in that: The hardware mapping is performed by using the 3D sparse convolution algorithm based on the NPU chip, specifically including: After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain sparse representation; After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain a collection table; According to the sparse representation and the collection table, the collected features are obtained, and the obtained collected features are multiplied by the expanded convolution kernel.

4. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 3, characterized in that: For sparse representation, specifically, after the 3D space scanned by the laser is divided into multiple Voxels, the entire Voxel is constructed into a sparse representation, and the directory of the sparse representation includes entries id, pos, and values, wherein the pos is the position of the Voxel in the 3D voxel space, and the values ​​is the feature vector of the Voxel, wherein the position where the data in the Voxel is not 0 is constructed as an entry in the sparse representation; For the collection table, specifically, the collection table is used to arrange the positions where convolution calculations need to be performed. The directory of the collection table includes a position id and a digital number. The position id represents the id of the position where convolution is required, and the digital number is the Voxel that needs to be calculated corresponding to each 3D convolution kernel.

5. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 3, characterized in that: When multiplying the collected features with the expanded convolution kernel: The dimension of the expanded convolution kernel is expressed as (IC * K) * OC, where IC is the input feature dimension of the convolution kernel of a certain layer, OC is the output feature dimension of the convolution kernel of a certain layer, and K is the number of 3D convolution kernels; The dimension of the collected features is expressed as N*IC*K, where N is the total length of the position id in the collection table.

6. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 1, characterized in that: The hardware mapping is performed through an end-to-end algorithm based on voxel or pillar features, specifically including: Send the point cloud to ASA to extract Voxel and Pillar features to obtain Voxel features and Pillar features; The Pillar features are fed into a 2D convolutional network for feature extraction, and the Voxel features are fed into a sparse 3D convolutional network for feature extraction; The features extracted by the sparse 3D convolutional network are sent to the Neck or Head based on the convolutional network, or to the Neck or Head based on the Transformer, and the output result is obtained and post-processed to obtain the final output of the end-to-end algorithm.

7. The ASIC hardware acceleration and end-to-end laser point cloud implementation method according to claim 6, characterized in that: When extracting Pillar features, the corresponding implementation hardware is NPU; When extracting Voxel features, the corresponding implementation hardware is NPU, DSP or SCU.

8. An ASIC hardware acceleration and end-to-end laser point cloud realization system, characterized in that: The ASIC hardware acceleration and end-to-end laser point cloud implementation system includes: A design module, which is used to design an ASIC for laser point cloud pre-processing coding to obtain a LiDAR point cloud ASA, wherein the LiDAR point cloud ASA includes hard voxelization, dynamic voxelization and voxel feature extraction; A first execution module, which is used to perform hardware mapping through a 3D sparse convolution algorithm based on an NPU chip; The second execution module is used to perform hardware mapping through an end-to-end algorithm based on voxel or pillar features, thereby achieving hardware acceleration and end-to-end laser point cloud realization.

9. The ASIC hardware acceleration and end-to-end laser point cloud realization system according to claim 8, characterized in that: The dynamic voxelization is a process of converting continuous point cloud data into discrete voxel representation; The hard voxelization is to aggregate the point cloud data into voxels based on the dynamic voxelization; The voxel feature extraction is to take points or padding, perform feature encoding on the point cloud, and AISC performs parallel calculations on different voxels or Pillars.

10. The ASIC hardware acceleration and end-to-end laser point cloud realization system according to claim 8, characterized in that: The hardware mapping is performed by using the 3D sparse convolution algorithm based on the NPU chip, specifically including: After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain sparse representation; After the point cloud is processed by ASA to extract features, it is processed by CPU, DSP or SCU to obtain a collection table; According to the sparse representation and the collection table, the collected features are obtained, and the obtained collected features are multiplied by the expanded convolution kernel.